[00:12:32] <wikibugs_>	 (03CR) 10BryanDavis: [V: 032 C: 031] "Tested using /usr/local/bin/sql.I038d14fb73a9bfe633003a6ab89d712510f61f61 on tools-bastion-03." [labs/toollabs] - 10https://gerrit.wikimedia.org/r/340233 (owner: 10Tim Landscheidt)
[00:19:36] <wikibugs_>	 06Labs, 10Tool-Labs: Restore replica.my.cnf for toolsbeta.admin - https://phabricator.wikimedia.org/T109807#3059871 (10scfc) @chasemp: Thanks for the explanation.  Running `maintain-dbusers` for several projects in parallel would indeed be problematic for user accounts.  I don't want to see this issue blocked...
[00:53:33] <wikibugs_>	 10Tool-Labs-tools-Xtools: Populate labs/tools/wikiviewstats git repo - https://phabricator.wikimedia.org/T108058#1511272 (10Krinkle) It seems <https://tools.wmflabs.org/wikiviewstats/> is broken:  > ( ! ) Fatal error: Call to a member function format() on a non-object in /mnt/nfs/labstore-secondary-tools-project...
[01:00:26] <wikibugs_>	 06Labs, 10Labs-Infrastructure: set apt to Wikimedia mirror instead of http://nova.clouds.archive.ubuntu.com/ubuntu/ - https://phabricator.wikimedia.org/T68121#3059966 (10scfc) 05Open>03declined a:05scfc>03None I still think this is worthwhile to pursue, but it's too "complicated" for my liking in the e...
[01:50:50] <wikibugs_>	 06Labs, 10Tool-Labs: Clean up cruft in urlproxy - https://phabricator.wikimedia.org/T95442#3060044 (10scfc) 05Open>03declined My understanding here was flawed: I assumed that the same Lua modules for for example Redis could be used for Resty as for the usual applications.  But Resty needs special coroutine...
[02:23:48] <wikibugs_>	 10Tool-Labs-tools-Other, 06Discovery, 10GeoData, 07Community-Wishlist-Survey-2016, 06Maps (Kartographer): Better interface and visualisation for coordinates and map - https://phabricator.wikimedia.org/T157844#3060085 (10srishakatux) (NOTE) This task was proposed in the #community-wishlist-survey-2016 and...
[02:25:26] <wikibugs_>	 10Tool-Labs-tools-Other, 06Commons, 06Community-Tech, 10Internet-Archive, 07Community-Wishlist-Survey-2016: Create a new DerivativeFX after the Toolserver shutdown [AOI] - https://phabricator.wikimedia.org/T110409#1577252 (10srishakatux) (NOTE) This task was proposed in the #community-wishlist-survey-201...
[02:26:04] <wikibugs_>	 06Labs: Request creation of Wikimedia Incubator labs project - https://phabricator.wikimedia.org/T159068#3060130 (10Hydriz) Apologies. If Parsoid can run behind the webproxy then I am fine with not having a public IP. Thanks for your help!
[02:56:21] <wikibugs_>	 06Labs, 10Labs-Sprint-101: Provide all labs users with username / passwords for the Postgres database - https://phabricator.wikimedia.org/T101661#3060157 (10scfc)
[02:56:24] <wikibugs_>	 06Labs, 06Operations: Backport python-ldap3 package from Utopic to Precise / Trusty - https://phabricator.wikimedia.org/T101824#3060155 (10scfc) 05Open>03declined >>! In T101824#1737022, @Dzahn wrote: > What's still missing?  Actually this issue is mostly obsolete.  While `python-ldap3` was backported to T...
[03:03:49] <wikibugs_>	 (03CR) 10Tim Landscheidt: [C: 032] Package jmail [labs/toollabs] - 10https://gerrit.wikimedia.org/r/339920 (https://phabricator.wikimedia.org/T158722) (owner: 10Tim Landscheidt)
[03:05:32] <wikibugs_>	 (03Merged) 10jenkins-bot: Package jmail [labs/toollabs] - 10https://gerrit.wikimedia.org/r/339920 (https://phabricator.wikimedia.org/T158722) (owner: 10Tim Landscheidt)
[03:09:22] <wikibugs_>	 06Labs, 10Labs-Sprint-101: Provide all labs users with username / passwords for the Postgres database - https://phabricator.wikimedia.org/T101661#3060164 (10scfc) p:05Normal>03Lowest (There does not seem to be anyone planning to work on this.)
[03:13:57] <wikibugs_>	 (03PS2) 10Tim Landscheidt: Port sql to Python [labs/toollabs] - 10https://gerrit.wikimedia.org/r/340233
[03:15:40] <wikibugs_>	 (03CR) 10Tim Landscheidt: [C: 032] Port sql to Python [labs/toollabs] - 10https://gerrit.wikimedia.org/r/340233 (owner: 10Tim Landscheidt)
[03:16:56] <wikibugs_>	 (03Merged) 10jenkins-bot: Port sql to Python [labs/toollabs] - 10https://gerrit.wikimedia.org/r/340233 (owner: 10Tim Landscheidt)
[03:20:23] <wikibugs_>	 (03PS1) 10Tim Landscheidt: Cut release 1.20 [labs/toollabs] - 10https://gerrit.wikimedia.org/r/340275
[03:21:51] <wikibugs_>	 (03CR) 10Tim Landscheidt: [C: 032] Cut release 1.20 [labs/toollabs] - 10https://gerrit.wikimedia.org/r/340275 (owner: 10Tim Landscheidt)
[03:23:00] <wikibugs_>	 (03Merged) 10jenkins-bot: Cut release 1.20 [labs/toollabs] - 10https://gerrit.wikimedia.org/r/340275 (owner: 10Tim Landscheidt)
[03:30:58] <wikibugs_>	 (03PS1) 10Tim Landscheidt: Package jmail [labs/toollabs] (ubuntu/precise) - 10https://gerrit.wikimedia.org/r/340276 (https://phabricator.wikimedia.org/T158722)
[03:31:00] <wikibugs_>	 (03PS1) 10Tim Landscheidt: Port sql to Python [labs/toollabs] (ubuntu/precise) - 10https://gerrit.wikimedia.org/r/340277
[03:31:02] <wikibugs_>	 (03PS1) 10Tim Landscheidt: Cut release 1.20~precise+1 [labs/toollabs] (ubuntu/precise) - 10https://gerrit.wikimedia.org/r/340278
[03:34:23] <wikibugs_>	 (03CR) 10Tim Landscheidt: [C: 032] Package jmail [labs/toollabs] (ubuntu/precise) - 10https://gerrit.wikimedia.org/r/340276 (https://phabricator.wikimedia.org/T158722) (owner: 10Tim Landscheidt)
[03:34:36] <wikibugs_>	 (03CR) 10Tim Landscheidt: [C: 032] Port sql to Python [labs/toollabs] (ubuntu/precise) - 10https://gerrit.wikimedia.org/r/340277 (owner: 10Tim Landscheidt)
[03:34:46] <wikibugs_>	 (03CR) 10Tim Landscheidt: [C: 032] Cut release 1.20~precise+1 [labs/toollabs] (ubuntu/precise) - 10https://gerrit.wikimedia.org/r/340278 (owner: 10Tim Landscheidt)
[03:35:39] <wikibugs_>	 (03Merged) 10jenkins-bot: Package jmail [labs/toollabs] (ubuntu/precise) - 10https://gerrit.wikimedia.org/r/340276 (https://phabricator.wikimedia.org/T158722) (owner: 10Tim Landscheidt)
[03:35:55] <wikibugs_>	 (03Merged) 10jenkins-bot: Port sql to Python [labs/toollabs] (ubuntu/precise) - 10https://gerrit.wikimedia.org/r/340277 (owner: 10Tim Landscheidt)
[03:36:44] <wikibugs_>	 (03Merged) 10jenkins-bot: Cut release 1.20~precise+1 [labs/toollabs] (ubuntu/precise) - 10https://gerrit.wikimedia.org/r/340278 (owner: 10Tim Landscheidt)
[03:50:22] <shinken-wm_>	 PROBLEM - Puppet run on tools-bastion-03 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0]
[03:52:27] <scfc_de>	 !log tools Deployed jobtools and misctools 1.20/1.20~precise+1 (T158722).
[03:52:34] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[03:52:34] <stashbot>	 T158722: jmail is not puppetized - https://phabricator.wikimedia.org/T158722
[03:58:00] <wikibugs_>	 06Labs, 10Tool-Labs, 13Patch-For-Review: jmail is not puppetized - https://phabricator.wikimedia.org/T158722#3060185 (10scfc) 05Open>03Resolved I removed the now-obsolete files under `/usr/local/bin` and tested successfully `/usr/bin/jmail` with `echo Test 15 | mail -s Test tools.scfc-test-can-be-deleted...
[04:25:25] <shinken-wm_>	 RECOVERY - Puppet run on tools-bastion-03 is OK: OK: Less than 1.00% above the threshold [0.0]
[05:12:47] <wikibugs_>	 06Labs, 10Tool-Labs, 10Tools-Kubernetes, 13Patch-For-Review: k8s webservice restart failure with `ValueError: get() more than one object; use filter` - https://phabricator.wikimedia.org/T156626#3060221 (10scfc) >>! In T156626#3029015, @JustBerry wrote: > @scfc No worries, but http://tools.wmflabs.org/spiar...
[06:37:04] <shinken-wm_>	 PROBLEM - Puppet run on tools-exec-1221 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0]
[06:52:11] <shinken-wm_>	 PROBLEM - Puppet run on tools-webgrid-lighttpd-1201 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0]
[06:53:48] <shinken-wm_>	 PROBLEM - Puppet run on tools-worker-1010 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0]
[06:53:54] <shinken-wm_>	 PROBLEM - Puppet run on tools-worker-1005 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0]
[06:54:36] <shinken-wm_>	 PROBLEM - Puppet run on tools-docker-registry-01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0]
[06:55:33] <shinken-wm_>	 PROBLEM - Puppet run on tools-docker-registry-02 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0]
[06:55:37] <shinken-wm_>	 PROBLEM - Puppet run on tools-webgrid-lighttpd-1407 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0]
[06:55:49] <shinken-wm_>	 PROBLEM - Puppet run on tools-webgrid-lighttpd-1404 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0]
[06:55:59] <shinken-wm_>	 PROBLEM - Puppet run on tools-worker-1008 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0]
[06:56:01] <shinken-wm_>	 PROBLEM - Puppet run on tools-exec-1415 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0]
[06:56:03] <shinken-wm_>	 PROBLEM - Puppet run on tools-exec-1218 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0]
[06:56:09] <shinken-wm_>	 PROBLEM - Puppet run on tools-webgrid-lighttpd-1414 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0]
[06:56:09] <shinken-wm_>	 PROBLEM - Puppet run on tools-webgrid-lighttpd-1402 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0]
[06:56:11] <shinken-wm_>	 PROBLEM - Puppet run on tools-webgrid-generic-1401 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0]
[06:56:11] <shinken-wm_>	 PROBLEM - Puppet run on tools-webgrid-lighttpd-1203 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0]
[06:56:13] <shinken-wm_>	 PROBLEM - Puppet run on tools-webgrid-lighttpd-1207 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0]
[06:56:55] <shinken-wm_>	 PROBLEM - Puppet run on tools-proxy-01 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0]
[06:57:13] <shinken-wm_>	 PROBLEM - Puppet run on tools-logs-02 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0]
[06:57:15] <shinken-wm_>	 PROBLEM - Puppet run on tools-redis-1001 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0]
[06:57:23] <shinken-wm_>	 PROBLEM - Puppet run on tools-exec-1219 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0]
[06:57:30] <shinken-wm_>	 PROBLEM - Puppet run on tools-webgrid-lighttpd-1406 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0]
[06:57:32] <shinken-wm_>	 PROBLEM - Puppet run on tools-webgrid-lighttpd-1208 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0]
[06:57:36] <shinken-wm_>	 PROBLEM - Puppet run on tools-webgrid-lighttpd-1204 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0]
[06:57:42] <shinken-wm_>	 PROBLEM - Puppet run on tools-exec-1404 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0]
[06:58:32] <shinken-wm_>	 PROBLEM - Puppet run on tools-static-10 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0]
[06:58:42] <shinken-wm_>	 PROBLEM - Puppet run on tools-exec-1408 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0]
[06:58:44] <shinken-wm_>	 PROBLEM - Puppet run on tools-exec-1405 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0]
[06:59:08] <shinken-wm_>	 PROBLEM - Puppet run on tools-exec-1419 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0]
[06:59:20] <shinken-wm_>	 PROBLEM - Puppet run on tools-worker-1019 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0]
[06:59:40] <shinken-wm_>	 PROBLEM - Puppet run on tools-worker-1022 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0]
[07:00:29] <shinken-wm_>	 PROBLEM - Puppet run on tools-worker-1014 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0]
[07:00:59] <shinken-wm_>	 PROBLEM - Puppet run on tools-worker-1003 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0]
[07:01:01] <shinken-wm_>	 PROBLEM - Puppet run on tools-webgrid-generic-1403 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0]
[07:01:11] <shinken-wm_>	 PROBLEM - Puppet run on tools-exec-1413 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0]
[07:02:18] <shinken-wm_>	 PROBLEM - Puppet run on tools-webgrid-lighttpd-1413 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0]
[07:02:21] <shinken-wm_>	 PROBLEM - Puppet run on tools-webgrid-generic-1404 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0]
[07:02:23] <shinken-wm_>	 PROBLEM - Puppet run on tools-worker-1004 is CRITICAL: CRITICAL: 70.00% of data above the critical threshold [0.0]
[07:03:02] <shinken-wm_>	 PROBLEM - Puppet run on tools-exec-1403 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0]
[07:03:18] <shinken-wm_>	 PROBLEM - Puppet run on tools-worker-1015 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0]
[07:03:32] <shinken-wm_>	 PROBLEM - Puppet run on tools-flannel-etcd-03 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0]
[07:03:46] <shinken-wm_>	 PROBLEM - Puppet run on tools-bastion-05 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0]
[07:03:46] <shinken-wm_>	 PROBLEM - Puppet run on tools-worker-1025 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0]
[07:03:50] <shinken-wm_>	 PROBLEM - Puppet run on tools-elastic-03 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0]
[07:04:14] <shinken-wm_>	 PROBLEM - Puppet run on tools-webgrid-lighttpd-1405 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0]
[07:04:41] <shinken-wm_>	 PROBLEM - Puppet run on tools-webgrid-lighttpd-1418 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0]
[07:04:49] <shinken-wm_>	 PROBLEM - Puppet run on tools-exec-1402 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0]
[07:05:07] <shinken-wm_>	 PROBLEM - Puppet run on tools-exec-1401 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0]
[07:05:39] <shinken-wm_>	 PROBLEM - Puppet run on tools-exec-1416 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0]
[07:06:35] <shinken-wm_>	 PROBLEM - Puppet run on tools-exec-1414 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0]
[07:06:55] <shinken-wm_>	 PROBLEM - Puppet run on tools-prometheus-02 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0]
[07:06:57] <shinken-wm_>	 PROBLEM - Puppet run on tools-flannel-etcd-01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0]
[07:07:04] <shinken-wm_>	 PROBLEM - Puppet run on tools-services-02 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0]
[07:07:42] <shinken-wm_>	 PROBLEM - Puppet run on tools-services-01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0]
[07:07:52] <shinken-wm_>	 PROBLEM - Puppet run on tools-worker-1013 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0]
[07:08:04] <shinken-wm_>	 PROBLEM - Puppet run on tools-puppetmaster-02 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0]
[07:08:08] <shinken-wm_>	 PROBLEM - Puppet run on tools-exec-1407 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0]
[07:08:14] <shinken-wm_>	 PROBLEM - Puppet run on tools-precise-dev is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0]
[07:08:30] <shinken-wm_>	 PROBLEM - Puppet run on tools-worker-1007 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0]
[07:08:43] <shinken-wm_>	 PROBLEM - Puppet run on tools-proxy-02 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0]
[07:09:08] <shinken-wm_>	 PROBLEM - Puppet run on tools-worker-1023 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0]
[07:09:10] <shinken-wm_>	 PROBLEM - Puppet run on tools-worker-1026 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0]
[07:09:35] <shinken-wm_>	 PROBLEM - Puppet run on tools-webgrid-lighttpd-1403 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0]
[07:09:37] <shinken-wm_>	 PROBLEM - Puppet run on tools-exec-1406 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0]
[07:10:13] <shinken-wm_>	 PROBLEM - Puppet run on tools-webgrid-lighttpd-1206 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0]
[07:10:25] <shinken-wm_>	 PROBLEM - Puppet run on tools-checker-02 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0]
[07:10:39] <shinken-wm_>	 PROBLEM - Puppet run on tools-exec-1417 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0]
[07:10:43] <shinken-wm_>	 PROBLEM - Puppet run on tools-exec-1411 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0]
[07:12:15] <shinken-wm_>	 PROBLEM - Puppet run on tools-redis-1002 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0]
[07:12:21] <shinken-wm_>	 PROBLEM - Puppet run on tools-prometheus-01 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0]
[07:12:29] <shinken-wm_>	 PROBLEM - Puppet run on tools-static-11 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0]
[07:12:44] <shinken-wm_>	 PROBLEM - Puppet run on tools-bastion-02 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0]
[07:12:54] <shinken-wm_>	 PROBLEM - Puppet run on tools-worker-1002 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0]
[07:13:12] <shinken-wm_>	 PROBLEM - Puppet run on tools-webgrid-lighttpd-1412 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0]
[07:13:18] <shinken-wm_>	 PROBLEM - Puppet run on tools-worker-1001 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0]
[07:13:46] <shinken-wm_>	 PROBLEM - Puppet run on tools-worker-1016 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0]
[07:13:58] <shinken-wm_>	 PROBLEM - Puppet run on tools-k8s-master-01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0]
[07:14:00] <shinken-wm_>	 PROBLEM - Puppet run on tools-webgrid-lighttpd-1408 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0]
[07:14:28] <shinken-wm_>	 PROBLEM - Puppet run on tools-worker-1017 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0]
[07:14:29] <shinken-wm_>	 PROBLEM - Puppet run on tools-exec-1409 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0]
[07:14:34] <shinken-wm_>	 PROBLEM - Puppet run on tools-flannel-etcd-02 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0]
[07:14:42] <shinken-wm_>	 PROBLEM - Puppet run on tools-exec-gift-trusty-01 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0]
[07:14:50] <shinken-wm_>	 PROBLEM - Puppet run on tools-webgrid-lighttpd-1409 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0]
[07:15:35] <shinken-wm_>	 PROBLEM - Puppet run on tools-k8s-etcd-01 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0]
[07:15:49] <shinken-wm_>	 PROBLEM - Puppet run on tools-exec-1217 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0]
[07:15:55] <shinken-wm_>	 PROBLEM - Puppet run on tools-exec-gift is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0]
[07:16:11] <shinken-wm_>	 PROBLEM - Puppet run on tools-webgrid-lighttpd-1210 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0]
[07:16:11] <shinken-wm_>	 PROBLEM - Puppet run on tools-webgrid-lighttpd-1209 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0]
[07:16:21] <shinken-wm_>	 PROBLEM - Puppet run on tools-bastion-03 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0]
[07:16:58] <shinken-wm_>	 PROBLEM - Puppet run on tools-exec-1418 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0]
[07:17:14] <shinken-wm_>	 PROBLEM - Puppet run on tools-webgrid-lighttpd-1205 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0]
[07:17:16] <shinken-wm_>	 PROBLEM - Puppet run on tools-worker-1009 is CRITICAL: CRITICAL: 62.50% of data above the critical threshold [0.0]
[07:17:20] <shinken-wm_>	 PROBLEM - Puppet run on tools-exec-1412 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0]
[07:17:22] <shinken-wm_>	 PROBLEM - Puppet run on tools-mail is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0]
[07:17:22] <shinken-wm_>	 PROBLEM - Puppet run on tools-webgrid-lighttpd-1416 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0]
[07:17:23] <shinken-wm_>	 PROBLEM - Puppet run on tools-exec-1410 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0]
[07:17:52] <shinken-wm_>	 PROBLEM - Puppet run on tools-exec-1220 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0]
[07:18:00] <shinken-wm_>	 PROBLEM - Puppet run on tools-worker-1012 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0]
[07:18:18] <shinken-wm_>	 PROBLEM - Puppet run on tools-elastic-01 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0]
[07:18:19] <shinken-wm_>	 PROBLEM - Puppet run on tools-worker-1006 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0]
[07:18:28] <shinken-wm_>	 PROBLEM - Puppet run on tools-worker-1018 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0]
[07:18:34] <shinken-wm_>	 PROBLEM - Puppet run on tools-k8s-etcd-02 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0]
[07:18:56] <shinken-wm_>	 PROBLEM - Puppet run on tools-webgrid-generic-1402 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0]
[07:19:08] <shinken-wm_>	 PROBLEM - Puppet run on tools-webgrid-lighttpd-1202 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0]
[07:19:11] <shinken-wm_>	 PROBLEM - Puppet run on tools-webgrid-lighttpd-1410 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0]
[07:19:13] <shinken-wm_>	 PROBLEM - Puppet run on tools-cron-01 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0]
[07:19:25] <shinken-wm_>	 PROBLEM - Puppet run on tools-worker-1011 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0]
[07:19:31] <shinken-wm_>	 PROBLEM - Puppet run on tools-docker-builder-04 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0]
[07:19:41] <shinken-wm_>	 PROBLEM - Puppet run on tools-webgrid-lighttpd-1415 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0]
[07:21:13] <shinken-wm_>	 PROBLEM - Puppet run on tools-exec-1420 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0]
[07:21:15] <shinken-wm_>	 PROBLEM - Puppet run on tools-elastic-02 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0]
[07:21:57] <shinken-wm_>	 PROBLEM - Puppet run on tools-worker-1020 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0]
[07:22:01] <shinken-wm_>	 PROBLEM - Puppet run on tools-grid-shadow is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0]
[07:22:35] <shinken-wm_>	 PROBLEM - Puppet run on tools-worker-1027 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0]
[07:23:08] <shinken-wm_>	 PROBLEM - Puppet run on tools-grid-master is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0]
[07:23:13] <shinken-wm_>	 PROBLEM - Puppet run on tools-checker-01 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0]
[07:23:25] <shinken-wm_>	 PROBLEM - Puppet run on tools-webgrid-lighttpd-1401 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0]
[07:23:38] <shinken-wm_>	 PROBLEM - Puppet run on tools-k8s-etcd-03 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0]
[07:24:12] <shinken-wm_>	 PROBLEM - Puppet run on tools-worker-1021 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0]
[07:24:24] <shinken-wm_>	 PROBLEM - Puppet run on tools-webgrid-lighttpd-1411 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0]
[09:28:26] <_joe_>	 puppetmaster on tools should be back now
[09:37:18] <wikibugs_>	 06Labs, 10wikitech.wikimedia.org: Can't reset password on wikitech (Unicode passwords not accepted), due to LDAP/opendj? - https://phabricator.wikimedia.org/T58114#3060493 (10Nemo_bis) After changing to another email address, I was able to get a password reset and the password "81L_['-nD:¬³ΩŁ¢®Ŧ¥ª§&JĦŊÐÄä±ºД№⌘...
[09:38:04] <shinken-wm_>	 RECOVERY - Puppet run on tools-puppetmaster-02 is OK: OK: Less than 1.00% above the threshold [0.0]
[09:39:49] <shinken-wm_>	 RECOVERY - Puppet run on tools-exec-1402 is OK: OK: Less than 1.00% above the threshold [0.0]
[09:40:29] <shinken-wm_>	 RECOVERY - Puppet run on tools-worker-1014 is OK: OK: Less than 1.00% above the threshold [0.0]
[09:40:57] <shinken-wm_>	 RECOVERY - Puppet run on tools-worker-1003 is OK: OK: Less than 1.00% above the threshold [0.0]
[09:41:11] <shinken-wm_>	 RECOVERY - Puppet run on tools-exec-1413 is OK: OK: Less than 1.00% above the threshold [0.0]
[09:42:05] <shinken-wm_>	 RECOVERY - Puppet run on tools-exec-1221 is OK: OK: Less than 1.00% above the threshold [0.0]
[09:42:21] <shinken-wm_>	 RECOVERY - Puppet run on tools-webgrid-generic-1404 is OK: OK: Less than 1.00% above the threshold [0.0]
[09:42:49] <_joe_>	 Nemo_bis: thanks for sharing your password with us! :P
[09:42:59] <shinken-wm_>	 RECOVERY - Puppet run on tools-exec-1403 is OK: OK: Less than 1.00% above the threshold [0.0]
[09:43:15] <shinken-wm_>	 RECOVERY - Puppet run on tools-precise-dev is OK: OK: Less than 1.00% above the threshold [0.0]
[09:43:29] <shinken-wm_>	 RECOVERY - Puppet run on tools-flannel-etcd-03 is OK: OK: Less than 1.00% above the threshold [0.0]
[09:43:31] <shinken-wm_>	 RECOVERY - Puppet run on tools-worker-1007 is OK: OK: Less than 1.00% above the threshold [0.0]
[09:43:47] <shinken-wm_>	 RECOVERY - Puppet run on tools-worker-1025 is OK: OK: Less than 1.00% above the threshold [0.0]
[09:43:50] <shinken-wm_>	 RECOVERY - Puppet run on tools-bastion-05 is OK: OK: Less than 1.00% above the threshold [0.0]
[09:43:50] <shinken-wm_>	 RECOVERY - Puppet run on tools-elastic-03 is OK: OK: Less than 1.00% above the threshold [0.0]
[09:44:08] <shinken-wm_>	 RECOVERY - Puppet run on tools-worker-1026 is OK: OK: Less than 1.00% above the threshold [0.0]
[09:44:12] <shinken-wm_>	 RECOVERY - Puppet run on tools-webgrid-lighttpd-1405 is OK: OK: Less than 1.00% above the threshold [0.0]
[09:44:42] <shinken-wm_>	 RECOVERY - Puppet run on tools-webgrid-lighttpd-1418 is OK: OK: Less than 1.00% above the threshold [0.0]
[09:45:06] <shinken-wm_>	 RECOVERY - Puppet run on tools-exec-1401 is OK: OK: Less than 1.00% above the threshold [0.0]
[09:45:14] <shinken-wm_>	 RECOVERY - Puppet run on tools-webgrid-lighttpd-1206 is OK: OK: Less than 1.00% above the threshold [0.0]
[09:45:40] <shinken-wm_>	 RECOVERY - Puppet run on tools-exec-1417 is OK: OK: Less than 1.00% above the threshold [0.0]
[09:45:41] <shinken-wm_>	 RECOVERY - Puppet run on tools-exec-1416 is OK: OK: Less than 1.00% above the threshold [0.0]
[09:45:42] <shinken-wm_>	 RECOVERY - Puppet run on tools-exec-1411 is OK: OK: Less than 1.00% above the threshold [0.0]
[09:46:34] <shinken-wm_>	 RECOVERY - Puppet run on tools-exec-1414 is OK: OK: Less than 1.00% above the threshold [0.0]
[09:46:56] <shinken-wm_>	 RECOVERY - Puppet run on tools-prometheus-02 is OK: OK: Less than 1.00% above the threshold [0.0]
[09:46:57] <shinken-wm_>	 RECOVERY - Puppet run on tools-flannel-etcd-01 is OK: OK: Less than 1.00% above the threshold [0.0]
[09:47:03] <shinken-wm_>	 RECOVERY - Puppet run on tools-services-02 is OK: OK: Less than 1.00% above the threshold [0.0]
[09:47:17] <shinken-wm_>	 RECOVERY - Puppet run on tools-redis-1002 is OK: OK: Less than 1.00% above the threshold [0.0]
[09:47:21] <shinken-wm_>	 RECOVERY - Puppet run on tools-prometheus-01 is OK: OK: Less than 1.00% above the threshold [0.0]
[09:47:29] <shinken-wm_>	 RECOVERY - Puppet run on tools-static-11 is OK: OK: Less than 1.00% above the threshold [0.0]
[09:47:43] <shinken-wm_>	 RECOVERY - Puppet run on tools-services-01 is OK: OK: Less than 1.00% above the threshold [0.0]
[09:47:51] <shinken-wm_>	 RECOVERY - Puppet run on tools-worker-1013 is OK: OK: Less than 1.00% above the threshold [0.0]
[09:47:53] <shinken-wm_>	 RECOVERY - Puppet run on tools-worker-1002 is OK: OK: Less than 1.00% above the threshold [0.0]
[09:48:07] <shinken-wm_>	 RECOVERY - Puppet run on tools-exec-1407 is OK: OK: Less than 1.00% above the threshold [0.0]
[09:48:17] <shinken-wm_>	 RECOVERY - Puppet run on tools-worker-1001 is OK: OK: Less than 1.00% above the threshold [0.0]
[09:48:43] <shinken-wm_>	 RECOVERY - Puppet run on tools-proxy-02 is OK: OK: Less than 1.00% above the threshold [0.0]
[09:49:07] <shinken-wm_>	 RECOVERY - Puppet run on tools-worker-1023 is OK: OK: Less than 1.00% above the threshold [0.0]
[09:49:32] <shinken-wm_>	 RECOVERY - Puppet run on tools-webgrid-lighttpd-1403 is OK: OK: Less than 1.00% above the threshold [0.0]
[09:49:38] <shinken-wm_>	 RECOVERY - Puppet run on tools-exec-1406 is OK: OK: Less than 1.00% above the threshold [0.0]
[09:50:29] <shinken-wm_>	 RECOVERY - Puppet run on tools-checker-02 is OK: OK: Less than 1.00% above the threshold [0.0]
[09:50:36] <shinken-wm_>	 RECOVERY - Puppet run on tools-k8s-etcd-01 is OK: OK: Less than 1.00% above the threshold [0.0]
[09:51:10] <shinken-wm_>	 RECOVERY - Puppet run on tools-webgrid-lighttpd-1210 is OK: OK: Less than 1.00% above the threshold [0.0]
[09:52:12] <shinken-wm_>	 RECOVERY - Puppet run on tools-webgrid-lighttpd-1205 is OK: OK: Less than 1.00% above the threshold [0.0]
[09:52:18] <shinken-wm_>	 RECOVERY - Puppet run on tools-worker-1009 is OK: OK: Less than 1.00% above the threshold [0.0]
[09:52:20] <shinken-wm_>	 RECOVERY - Puppet run on tools-exec-1410 is OK: OK: Less than 1.00% above the threshold [0.0]
[09:52:20] <shinken-wm_>	 RECOVERY - Puppet run on tools-mail is OK: OK: Less than 1.00% above the threshold [0.0]
[09:52:42] <shinken-wm_>	 RECOVERY - Puppet run on tools-bastion-02 is OK: OK: Less than 1.00% above the threshold [0.0]
[09:53:10] <shinken-wm_>	 RECOVERY - Puppet run on tools-webgrid-lighttpd-1412 is OK: OK: Less than 1.00% above the threshold [0.0]
[09:53:20] <shinken-wm_>	 RECOVERY - Puppet run on tools-elastic-01 is OK: OK: Less than 1.00% above the threshold [0.0]
[09:53:30] <shinken-wm_>	 RECOVERY - Puppet run on tools-worker-1018 is OK: OK: Less than 1.00% above the threshold [0.0]
[09:53:34] <shinken-wm_>	 RECOVERY - Puppet run on tools-k8s-etcd-02 is OK: OK: Less than 1.00% above the threshold [0.0]
[09:53:42] <shinken-wm_>	 RECOVERY - Puppet run on tools-worker-1016 is OK: OK: Less than 1.00% above the threshold [0.0]
[09:53:58] <shinken-wm_>	 RECOVERY - Puppet run on tools-k8s-master-01 is OK: OK: Less than 1.00% above the threshold [0.0]
[09:54:01] <shinken-wm_>	 RECOVERY - Puppet run on tools-webgrid-lighttpd-1408 is OK: OK: Less than 1.00% above the threshold [0.0]
[09:54:11] <shinken-wm_>	 RECOVERY - Puppet run on tools-webgrid-lighttpd-1410 is OK: OK: Less than 1.00% above the threshold [0.0]
[09:54:25] <shinken-wm_>	 RECOVERY - Puppet run on tools-worker-1011 is OK: OK: Less than 1.00% above the threshold [0.0]
[09:54:29] <shinken-wm_>	 RECOVERY - Puppet run on tools-exec-1409 is OK: OK: Less than 1.00% above the threshold [0.0]
[09:54:29] <shinken-wm_>	 RECOVERY - Puppet run on tools-worker-1017 is OK: OK: Less than 1.00% above the threshold [0.0]
[09:54:33] <shinken-wm_>	 RECOVERY - Puppet run on tools-flannel-etcd-02 is OK: OK: Less than 1.00% above the threshold [0.0]
[09:54:41] <shinken-wm_>	 RECOVERY - Puppet run on tools-exec-gift-trusty-01 is OK: OK: Less than 1.00% above the threshold [0.0]
[09:54:55] <shinken-wm_>	 RECOVERY - Puppet run on tools-webgrid-lighttpd-1409 is OK: OK: Less than 1.00% above the threshold [0.0]
[09:55:51] <shinken-wm_>	 RECOVERY - Puppet run on tools-exec-1217 is OK: OK: Less than 1.00% above the threshold [0.0]
[09:55:55] <shinken-wm_>	 RECOVERY - Puppet run on tools-exec-gift is OK: OK: Less than 1.00% above the threshold [0.0]
[09:56:11] <shinken-wm_>	 RECOVERY - Puppet run on tools-webgrid-lighttpd-1209 is OK: OK: Less than 1.00% above the threshold [0.0]
[09:56:24] <shinken-wm_>	 RECOVERY - Puppet run on tools-bastion-03 is OK: OK: Less than 1.00% above the threshold [0.0]
[09:56:58] <shinken-wm_>	 RECOVERY - Puppet run on tools-exec-1418 is OK: OK: Less than 1.00% above the threshold [0.0]
[09:57:00] <shinken-wm_>	 RECOVERY - Puppet run on tools-worker-1020 is OK: OK: Less than 1.00% above the threshold [0.0]
[09:57:02] <shinken-wm_>	 RECOVERY - Puppet run on tools-grid-shadow is OK: OK: Less than 1.00% above the threshold [0.0]
[09:57:20] <shinken-wm_>	 RECOVERY - Puppet run on tools-webgrid-lighttpd-1416 is OK: OK: Less than 1.00% above the threshold [0.0]
[09:57:20] <shinken-wm_>	 RECOVERY - Puppet run on tools-exec-1412 is OK: OK: Less than 1.00% above the threshold [0.0]
[09:57:52] <shinken-wm_>	 RECOVERY - Puppet run on tools-exec-1220 is OK: OK: Less than 1.00% above the threshold [0.0]
[09:58:00] <shinken-wm_>	 RECOVERY - Puppet run on tools-worker-1012 is OK: OK: Less than 1.00% above the threshold [0.0]
[09:58:08] <shinken-wm_>	 RECOVERY - Puppet run on tools-grid-master is OK: OK: Less than 1.00% above the threshold [0.0]
[09:58:14] <shinken-wm_>	 RECOVERY - Puppet run on tools-checker-01 is OK: OK: Less than 1.00% above the threshold [0.0]
[09:58:18] <shinken-wm_>	 RECOVERY - Puppet run on tools-worker-1006 is OK: OK: Less than 1.00% above the threshold [0.0]
[09:58:56] <shinken-wm_>	 RECOVERY - Puppet run on tools-webgrid-generic-1402 is OK: OK: Less than 1.00% above the threshold [0.0]
[09:59:08] <shinken-wm_>	 RECOVERY - Puppet run on tools-webgrid-lighttpd-1202 is OK: OK: Less than 1.00% above the threshold [0.0]
[09:59:12] <shinken-wm_>	 RECOVERY - Puppet run on tools-cron-01 is OK: OK: Less than 1.00% above the threshold [0.0]
[09:59:14] <shinken-wm_>	 RECOVERY - Puppet run on tools-worker-1021 is OK: OK: Less than 1.00% above the threshold [0.0]
[09:59:22] <shinken-wm_>	 RECOVERY - Puppet run on tools-webgrid-lighttpd-1411 is OK: OK: Less than 1.00% above the threshold [0.0]
[09:59:32] <shinken-wm_>	 RECOVERY - Puppet run on tools-docker-builder-04 is OK: OK: Less than 1.00% above the threshold [0.0]
[09:59:38] <shinken-wm_>	 RECOVERY - Puppet run on tools-webgrid-lighttpd-1415 is OK: OK: Less than 1.00% above the threshold [0.0]
[10:00:31] <shinken-wm_>	 RECOVERY - Puppet run on tools-docker-registry-02 is OK: OK: Less than 1.00% above the threshold [0.0]
[10:01:01] <shinken-wm_>	 RECOVERY - Puppet run on tools-worker-1008 is OK: OK: Less than 1.00% above the threshold [0.0]
[10:01:03] <shinken-wm_>	 RECOVERY - Puppet run on tools-exec-1218 is OK: OK: Less than 1.00% above the threshold [0.0]
[10:01:07] <shinken-wm_>	 RECOVERY - Puppet run on tools-webgrid-lighttpd-1402 is OK: OK: Less than 1.00% above the threshold [0.0]
[10:01:11] <shinken-wm_>	 RECOVERY - Puppet run on tools-webgrid-lighttpd-1203 is OK: OK: Less than 1.00% above the threshold [0.0]
[10:01:13] <shinken-wm_>	 RECOVERY - Puppet run on tools-exec-1420 is OK: OK: Less than 1.00% above the threshold [0.0]
[10:01:16] <shinken-wm_>	 RECOVERY - Puppet run on tools-elastic-02 is OK: OK: Less than 1.00% above the threshold [0.0]
[10:02:09] <shinken-wm_>	 RECOVERY - Puppet run on tools-webgrid-lighttpd-1201 is OK: OK: Less than 1.00% above the threshold [0.0]
[10:02:15] <shinken-wm_>	 RECOVERY - Puppet run on tools-logs-02 is OK: OK: Less than 1.00% above the threshold [0.0]
[10:02:16] <shinken-wm_>	 RECOVERY - Puppet run on tools-redis-1001 is OK: OK: Less than 1.00% above the threshold [0.0]
[10:02:22] <shinken-wm_>	 RECOVERY - Puppet run on tools-exec-1219 is OK: OK: Less than 1.00% above the threshold [0.0]
[10:02:34] <shinken-wm_>	 RECOVERY - Puppet run on tools-worker-1027 is OK: OK: Less than 1.00% above the threshold [0.0]
[10:02:38] <shinken-wm_>	 RECOVERY - Puppet run on tools-webgrid-lighttpd-1204 is OK: OK: Less than 1.00% above the threshold [0.0]
[10:03:24] <shinken-wm_>	 RECOVERY - Puppet run on tools-webgrid-lighttpd-1401 is OK: OK: Less than 1.00% above the threshold [0.0]
[10:03:33] <Nemo_bis>	 _joe_: yeah, now I'll go switch all my accounts to it ;) 
[10:03:38] <shinken-wm_>	 RECOVERY - Puppet run on tools-exec-1408 is OK: OK: Less than 1.00% above the threshold [0.0]
[10:03:38] <shinken-wm_>	 RECOVERY - Puppet run on tools-k8s-etcd-03 is OK: OK: Less than 1.00% above the threshold [0.0]
[10:03:42] <shinken-wm_>	 RECOVERY - Puppet run on tools-exec-1405 is OK: OK: Less than 1.00% above the threshold [0.0]
[10:03:55] <shinken-wm_>	 RECOVERY - Puppet run on tools-worker-1010 is OK: OK: Less than 1.00% above the threshold [0.0]
[10:03:55] <shinken-wm_>	 RECOVERY - Puppet run on tools-worker-1005 is OK: OK: Less than 1.00% above the threshold [0.0]
[10:04:40] <shinken-wm_>	 RECOVERY - Puppet run on tools-worker-1022 is OK: OK: Less than 1.00% above the threshold [0.0]
[10:04:41] <shinken-wm_>	 RECOVERY - Puppet run on tools-docker-registry-01 is OK: OK: Less than 1.00% above the threshold [0.0]
[10:05:39] <shinken-wm_>	 RECOVERY - Puppet run on tools-webgrid-lighttpd-1407 is OK: OK: Less than 1.00% above the threshold [0.0]
[10:05:49] <shinken-wm_>	 RECOVERY - Puppet run on tools-webgrid-lighttpd-1404 is OK: OK: Less than 1.00% above the threshold [0.0]
[10:06:01] <shinken-wm_>	 RECOVERY - Puppet run on tools-exec-1415 is OK: OK: Less than 1.00% above the threshold [0.0]
[10:06:07] <shinken-wm_>	 RECOVERY - Puppet run on tools-webgrid-lighttpd-1414 is OK: OK: Less than 1.00% above the threshold [0.0]
[10:06:13] <shinken-wm_>	 RECOVERY - Puppet run on tools-webgrid-generic-1401 is OK: OK: Less than 1.00% above the threshold [0.0]
[10:06:13] <shinken-wm_>	 RECOVERY - Puppet run on tools-webgrid-lighttpd-1207 is OK: OK: Less than 1.00% above the threshold [0.0]
[10:06:55] <shinken-wm_>	 RECOVERY - Puppet run on tools-proxy-01 is OK: OK: Less than 1.00% above the threshold [0.0]
[10:07:17] <shinken-wm_>	 RECOVERY - Puppet run on tools-webgrid-lighttpd-1413 is OK: OK: Less than 1.00% above the threshold [0.0]
[10:07:21] <shinken-wm_>	 RECOVERY - Puppet run on tools-worker-1004 is OK: OK: Less than 1.00% above the threshold [0.0]
[10:07:29] <shinken-wm_>	 RECOVERY - Puppet run on tools-webgrid-lighttpd-1406 is OK: OK: Less than 1.00% above the threshold [0.0]
[10:07:31] <shinken-wm_>	 RECOVERY - Puppet run on tools-webgrid-lighttpd-1208 is OK: OK: Less than 1.00% above the threshold [0.0]
[10:07:41] <shinken-wm_>	 RECOVERY - Puppet run on tools-exec-1404 is OK: OK: Less than 1.00% above the threshold [0.0]
[10:08:18] <shinken-wm_>	 RECOVERY - Puppet run on tools-worker-1015 is OK: OK: Less than 1.00% above the threshold [0.0]
[10:08:30] <shinken-wm_>	 RECOVERY - Puppet run on tools-static-10 is OK: OK: Less than 1.00% above the threshold [0.0]
[10:09:08] <shinken-wm_>	 RECOVERY - Puppet run on tools-exec-1419 is OK: OK: Less than 1.00% above the threshold [0.0]
[10:09:20] <shinken-wm_>	 RECOVERY - Puppet run on tools-worker-1019 is OK: OK: Less than 1.00% above the threshold [0.0]
[10:11:03] <shinken-wm_>	 RECOVERY - Puppet run on tools-webgrid-generic-1403 is OK: OK: Less than 1.00% above the threshold [0.0]
[12:22:39] <tarrow>	 Can anyone give me some advice on puppet on labs? I enable "role::puppetmaster::standalone". and on one instance this worked fine. On a new clean m.small trusty with only that role I get "Connection refused - connect(2) Could not retrieve file metadata for puppet://puppetmaster-02.wikifactmine.eqiad.wmflabs/plugins: Connection refused - connect(2)" I have
[12:22:39] <tarrow>	 puppetmaster set to the fqdn in Hiera and I tried with and without a security group opening port 8140
[13:19:47] <wikibugs_>	 (03PS1) 10Hashar: Dummy secret nova/osstackcanary [labs/private] - 10https://gerrit.wikimedia.org/r/340309
[13:21:46] <chasemp>	 tarrow: did you see https://wikitech.wikimedia.org/wiki/Standalone_puppetmaster#Step_2:_Setup_a_puppet_client
[16:18:53] <wikibugs_>	 06Labs, 10WM-Bot: Move wm-bot instance to Trusty - https://phabricator.wikimedia.org/T157838#3018288 (10Niedzielski) I think this is expected but in case it's not, no new content for #wikimedia-mobile has been logged for 10+ days: http://bots.wmflabs.org/~wm-bot/logs/%23wikimedia-mobile/
[16:26:21] <wikibugs_>	 06Labs, 10Tool-Labs, 10Tools-Kubernetes: Reassign service/pod IP ranges for kubernetes on tool labs - https://phabricator.wikimedia.org/T152399#3061504 (10chasemp) a:03chasemp
[16:30:49] <tarrow>	 chasemp:  that is what I was following but I got that error on step 1 part 3. I'm retrying on a totally new instance. 
[16:47:35] <chasemp>	 step 1 doesn't have a part 3 :)
[16:51:47] <wikibugs_>	 06Labs, 10Tool-Labs, 10Tools-Kubernetes: Reassign service/pod IP ranges for kubernetes on tool labs - https://phabricator.wikimedia.org/T152399#3061610 (10chasemp) https://etherpad.wikimedia.org/p/k8s-tools-renumber
[16:52:26] <wikibugs_>	 06Labs, 10Tool-Labs, 10Tools-Kubernetes: Reassign service/pod IP ranges for kubernetes on tool labs - https://phabricator.wikimedia.org/T152399#3061612 (10chasemp)
[16:54:38] <tarrow>	 "Force two puppet runs on the instance sudo puppet agent -t"
[16:56:04] <chasemp>	 ah I linked to teh client only setup, I didn't understand your issue on the standalone instance itself
[17:01:31] <tarrow>	 ah; yep. I'm slowly figuring it out :). I think the problem was I had a stale "puppetmaster" setting in Hiera. I'm trying again.
[17:02:44] <chasemp>	 ok, I have to afk for a bit but drop a note if things don't work out
[17:08:17] <tarrow>	 thanks :)
[17:43:12] <wikibugs_>	 06Labs, 10wikitech.wikimedia.org: Can't reset password on wikitech (Unicode passwords not accepted), due to LDAP/opendj? - https://phabricator.wikimedia.org/T58114#3061765 (10scfc) 05Open>03Resolved I think that is close enough to a verification.  2FA should not interfere with the passwords themselves.  Th...
[18:22:18] <wikibugs_>	 06Labs: Blacklist apache from unattended-upgrades on tools puppetmaster - https://phabricator.wikimedia.org/T159254#3061896 (10madhuvishy)
[18:22:30] <wikibugs_>	 06Labs: Blacklist apache from unattended-upgrades on tools puppetmaster - https://phabricator.wikimedia.org/T159254#3061908 (10madhuvishy) p:05Triage>03Normal
[18:27:24] <wm-bot>	 Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/WiktCAPT was created, changed by WiktCAPT link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/WiktCAPT edit summary: Created page with "{{Tools Access Request |Justification=to build back end leaderboard score and learner outreach support for a pocketsphinx.js-based Computer Aided Pronunciation Training tool t..."
[18:43:35] <wikibugs_>	 06Labs: Delete test-spm-1.project-proxy - https://phabricator.wikimedia.org/T159257#3061977 (10scfc)
[19:38:05] <wikibugs_>	 06Labs, 10Tool-Labs, 10Tools-Kubernetes, 13Patch-For-Review: k8s webservice restart failure with `ValueError: get() more than one object; use filter` - https://phabricator.wikimedia.org/T156626#3062118 (10scfc) a:05scfc>03None (Unassigning because I don't understand the failure mode, or rather, if my u...
[19:57:04] <wm-bot>	 Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/WiktCAPT was modified, changed by Tim Landscheidt link https://wikitech.wikimedia.org/w/index.php?diff=1605626 edit summary: 
[20:14:47] <wikibugs_>	 06Labs: Blacklist apache from unattended-upgrades on tools puppetmaster - https://phabricator.wikimedia.org/T159254#3061896 (10scfc) (FTR: To fix other instances with this problem (= every host with `apache2` running), `apt-get install apache2` is enough.)
[21:27:05] <wikibugs_>	 06Labs: Delete test-spm-1.project-proxy - https://phabricator.wikimedia.org/T159257#3062607 (10scfc) (The connection refusal is probably T159254 and could be fixed by `sudo apt-get install apache2`.)
[21:32:55] <wikibugs_>	 06Labs, 10Labs-Infrastructure, 13Patch-For-Review: Deprecate precise instances in Labs by 03/31/2017 - https://phabricator.wikimedia.org/T143349#2962987 (10Dzahn) Can i have a review for these?  https://gerrit.wikimedia.org/r/#/c/337207/ (drop icinga monitoring check called " tools-checker-grid-start-precise...
[21:44:05] <wikibugs_>	 06Labs, 07LDAP: Difficult to set up mock LDAP server in Labs - https://phabricator.wikimedia.org/T159277#3062676 (10scfc)
[21:56:09] <wikibugs_>	 06Labs, 10Labs-Infrastructure, 13Patch-For-Review: Deprecate precise instances in Labs by 03/31/2017 - https://phabricator.wikimedia.org/T143349#3062791 (10chasemp) >>! In T143349#3062635, @Dzahn wrote: > Can i have a review for these? >  > https://gerrit.wikimedia.org/r/#/c/337207/ (drop icinga monitoring c...
[22:09:04] <paladox>	 Hi, im sundanly getting these errord
[22:09:06] <paladox>	 errors
[22:09:10] <paladox>	 on my puppetmaster
[22:09:11] <paladox>	 Error: Could not retrieve catalog from remote server: Connection refused - connect(2) for "puppet-paladox.git.eqiad.wmflabs" port 8140
[22:09:12] <paladox>	 Warning: Not using cache on failed catalog
[22:09:12] <paladox>	 Error: Could not retrieve catalog; skipping run
[22:09:12] <paladox>	 Error: Could not send report: Connection refused - connect(2) for "puppet-paladox.git.eqiad.wmflabs" port 8140
[22:09:17] <paladox>	 It was working a day ago
[22:09:26] <paladox>	 but (928 minutes ago). it broke
[22:10:21] <paladox>	 seems to be working now
[22:10:28] <paladox>	 after restarting puppet and puppetmaster
[22:10:57] <paladox>	 it seems to be crashing
[22:11:05] <paladox>	 chasemp andrewbogott ^^
[22:11:22] <paladox>	 Feb 28 22:09:42 puppet-paladox systemd[1]: Starting Puppet agent...
[22:11:23] <paladox>	 Feb 28 22:09:44 puppet-paladox puppet-agent[10227]: Setting templatedir is deprecated. See http://links.puppetlabs.com/env-settings-deprecations
[22:11:23] <paladox>	 Feb 28 22:09:44 puppet-paladox puppet-agent[10227]: (at /usr/lib/ruby/vendor_ruby/puppet/settings.rb:1139:in `issue_deprecation_warning')
[22:11:23] <paladox>	 Feb 28 22:09:47 puppet-paladox puppet-agent[10364]: Reopening log files
[22:11:24] <paladox>	 Feb 28 22:09:47 puppet-paladox systemd[1]: Started Puppet agent.
[22:11:24] <paladox>	 Feb 28 22:09:47 puppet-paladox puppet-agent[10364]: Starting Puppet client version 3.8.5
[22:11:25] <paladox>	 Feb 28 22:10:37 puppet-paladox systemd[1]: Stopping Puppet agent...
[22:11:26] <paladox>	 Feb 28 22:10:37 puppet-paladox puppet-agent[10364]: Caught TERM; exiting
[22:11:27] <paladox>	 Feb 28 22:10:37 puppet-paladox systemd[1]: Stopped Puppet agent.
[22:11:37] <chasemp>	 paladox: probably https://phabricator.wikimedia.org/T159254
[22:11:52] <paladox>	 oh
[22:13:51] <paladox>	 thanks it works now
[22:13:53] <paladox>	 after upgrading apache
[22:14:25] <chasemp>	 np, that's probably worth an email to labs-announce when I can get a second
[22:16:29] <paladox>	 Yep :)
[22:56:08] <shinken-wm_>	 PROBLEM - Free space - all mounts on tools-proxy-01 is CRITICAL: CRITICAL: tools.tools-proxy-01.diskspace._public_dumps.byte_percentfree (No valid datapoints found)tools.tools-proxy-01.diskspace.root.byte_percentfree (<22.22%)
[23:16:06] <shinken-wm_>	 RECOVERY - Free space - all mounts on tools-proxy-01 is OK: OK: tools.tools-proxy-01.diskspace._public_dumps.byte_percentfree (No valid datapoints found)
[23:34:49] <wikibugs_>	 10Labs-project-Wikistats: miraheze custom domains not updated on wikistats - https://phabricator.wikimedia.org/T158976#3063167 (10Dzahn) I have manually started the script that cron should start to debug this. It found 1137 wikis, is then updating 1109 wikis.  Is that the difference and the number of custom wiki...