[00:00:35] PROBLEM HTTP is now: WARNING on deployment-apache21 i-0000026d output: HTTP WARNING: HTTP/1.1 403 Forbidden - 366 bytes in 0.004 second response time [00:29:46] RECOVERY Free ram is now: OK on ganglia-test2 i-00000250 output: OK: 85% free memory [02:42:21] 05/29/2012 - 02:42:21 - Updating keys for laner at /export/home/deployment-prep/laner [02:48:19] 05/29/2012 - 02:48:19 - Updating keys for laner at /export/home/deployment-prep/laner [02:49:19] 05/29/2012 - 02:49:19 - Updating keys for laner at /export/home/deployment-prep/laner [03:05:37] PROBLEM Current Load is now: CRITICAL on nagios 127.0.0.1 output: CRITICAL - load average: 12.03, 10.71, 6.12 [03:06:20] 05/29/2012 - 03:06:19 - Updating keys for laner at /export/home/deployment-prep/laner [03:08:24] 05/29/2012 - 03:08:24 - Updating keys for laner at /export/home/deployment-prep/laner [03:09:00] PROBLEM Current Load is now: WARNING on bots-cb i-0000009e output: WARNING - load average: 3.31, 10.45, 7.31 [03:19:38] RECOVERY Current Load is now: OK on bots-cb i-0000009e output: OK - load average: 0.37, 2.08, 4.29 [03:25:09] PROBLEM Current Load is now: WARNING on nagios 127.0.0.1 output: WARNING - load average: 0.90, 2.12, 3.96 [03:26:04] New patchset: Andrew Bogott; "Yet another stab at correct puppet syntax." [operations/puppet] (test) - https://gerrit.wikimedia.org/r/9213 [03:26:19] New review: gerrit2; "Lint check passed." [operations/puppet] (test); V: 1 - https://gerrit.wikimedia.org/r/9213 [03:27:00] New review: Andrew Bogott; "(no comment)" [operations/puppet] (test); V: 1 C: 2; - https://gerrit.wikimedia.org/r/9213 [03:27:02] Change merged: Andrew Bogott; [operations/puppet] (test) - https://gerrit.wikimedia.org/r/9213 [03:30:13] RECOVERY Current Load is now: OK on nagios 127.0.0.1 output: OK - load average: 0.54, 1.04, 2.96 [03:34:58] PROBLEM Free ram is now: WARNING on utils-abogott i-00000131 output: Warning: 17% free memory [03:44:20] 05/29/2012 - 03:44:20 - Updating keys for laner at /export/home/deployment-prep/laner [03:45:57] PROBLEM Free ram is now: WARNING on test-oneiric i-00000187 output: Warning: 13% free memory [03:50:01] PROBLEM Current Users is now: CRITICAL on maps-tilemill1 i-00000294 output: CHECK_NRPE: Socket timeout after 10 seconds. [03:51:18] PROBLEM Disk Space is now: CRITICAL on maps-tilemill1 i-00000294 output: CHECK_NRPE: Socket timeout after 10 seconds. [03:54:27] PROBLEM dpkg-check is now: CRITICAL on mwreview-test2 i-00000298 output: Connection refused by host [03:55:29] RECOVERY Current Users is now: OK on maps-tilemill1 i-00000294 output: USERS OK - 0 users currently logged in [03:56:05] PROBLEM Current Load is now: CRITICAL on mwreview-test2 i-00000298 output: Connection refused by host [03:56:14] PROBLEM Free ram is now: CRITICAL on utils-abogott i-00000131 output: Critical: 4% free memory [03:56:19] PROBLEM Current Load is now: CRITICAL on worker1 i-00000208 output: CHECK_NRPE: Socket timeout after 10 seconds. [03:56:19] PROBLEM Current Users is now: CRITICAL on worker1 i-00000208 output: CHECK_NRPE: Socket timeout after 10 seconds. [03:56:36] RECOVERY Disk Space is now: OK on maps-tilemill1 i-00000294 output: DISK OK [03:56:49] PROBLEM Current Users is now: CRITICAL on mwreview-test2 i-00000298 output: CHECK_NRPE: Socket timeout after 10 seconds. [03:56:54] PROBLEM Disk Space is now: CRITICAL on mwreview-test2 i-00000298 output: CHECK_NRPE: Socket timeout after 10 seconds. [03:57:28] PROBLEM Current Load is now: CRITICAL on incubator-bot1 i-00000251 output: CHECK_NRPE: Socket timeout after 10 seconds. [03:57:45] PROBLEM Free ram is now: CRITICAL on mwreview-test2 i-00000298 output: Connection refused by host [03:58:40] PROBLEM Total Processes is now: CRITICAL on mwreview-test2 i-00000298 output: Connection refused by host [04:00:30] RECOVERY Current Load is now: OK on worker1 i-00000208 output: OK - load average: 0.16, 1.92, 1.59 [04:00:30] RECOVERY Current Users is now: OK on worker1 i-00000208 output: USERS OK - 0 users currently logged in [04:00:30] RECOVERY Free ram is now: OK on utils-abogott i-00000131 output: OK: 96% free memory [04:01:00] PROBLEM Free ram is now: WARNING on nova-daas-1 i-000000e7 output: Warning: 14% free memory [04:01:30] PROBLEM Free ram is now: CRITICAL on test-oneiric i-00000187 output: Critical: 3% free memory [04:01:50] RECOVERY Current Load is now: OK on incubator-bot1 i-00000251 output: OK - load average: 0.28, 1.66, 1.49 [04:03:06] Eh, anyone knows how can I properly change my email address? [04:04:20] PROBLEM dpkg-check is now: UNKNOWN on mwreview-test2 i-00000298 output: Invalid host name i-00000298 [04:04:50] PROBLEM Free ram is now: WARNING on orgcharts-dev i-0000018f output: Warning: 13% free memory [04:06:34] RECOVERY Free ram is now: OK on test-oneiric i-00000187 output: OK: 97% free memory [04:09:11] PROBLEM host: mwreview-test2 is DOWN address: i-00000298 check_ping: Invalid hostname/address - i-00000298 [04:17:24] PROBLEM Total Processes is now: CRITICAL on mwreview-test3 i-00000299 output: Connection refused by host [04:17:34] PROBLEM dpkg-check is now: CRITICAL on mwreview-test3 i-00000299 output: Connection refused by host [04:20:19] PROBLEM Free ram is now: CRITICAL on orgcharts-dev i-0000018f output: Critical: 5% free memory [04:20:29] PROBLEM Current Load is now: CRITICAL on mwreview-test3 i-00000299 output: CHECK_NRPE: Socket timeout after 10 seconds. [04:21:07] PROBLEM Disk Space is now: CRITICAL on mwreview-test3 i-00000299 output: Connection refused or timed out [04:21:07] PROBLEM Current Users is now: CRITICAL on mwreview-test3 i-00000299 output: Connection refused or timed out [04:21:17] PROBLEM Free ram is now: CRITICAL on mwreview-test3 i-00000299 output: Connection refused by host [04:26:20] PROBLEM Disk Space is now: CRITICAL on maps-tilemill1 i-00000294 output: CHECK_NRPE: Socket timeout after 10 seconds. [04:26:25] RECOVERY Free ram is now: OK on orgcharts-dev i-0000018f output: OK: 94% free memory [04:26:25] PROBLEM Free ram is now: CRITICAL on nova-daas-1 i-000000e7 output: Critical: 5% free memory [04:27:58] PROBLEM Total Processes is now: CRITICAL on maps-tilemill1 i-00000294 output: CHECK_NRPE: Socket timeout after 10 seconds. [04:28:05] PROBLEM Current Load is now: CRITICAL on maps-tilemill1 i-00000294 output: CHECK_NRPE: Socket timeout after 10 seconds. [04:28:05] PROBLEM Free ram is now: CRITICAL on maps-tilemill1 i-00000294 output: CHECK_NRPE: Socket timeout after 10 seconds. [04:28:05] PROBLEM dpkg-check is now: CRITICAL on maps-tilemill1 i-00000294 output: CHECK_NRPE: Socket timeout after 10 seconds. [04:31:39] PROBLEM Current Users is now: CRITICAL on maps-tilemill1 i-00000294 output: CHECK_NRPE: Socket timeout after 10 seconds. [04:31:49] PROBLEM Current Users is now: CRITICAL on reportcard2 i-000001ea output: CHECK_NRPE: Socket timeout after 10 seconds. [04:31:49] PROBLEM Disk Space is now: CRITICAL on reportcard2 i-000001ea output: CHECK_NRPE: Socket timeout after 10 seconds. [04:33:09] PROBLEM Current Load is now: CRITICAL on worker1 i-00000208 output: CHECK_NRPE: Socket timeout after 10 seconds. [04:33:09] PROBLEM Current Users is now: CRITICAL on worker1 i-00000208 output: CHECK_NRPE: Socket timeout after 10 seconds. [04:33:14] PROBLEM Total Processes is now: CRITICAL on worker1 i-00000208 output: CHECK_NRPE: Socket timeout after 10 seconds. [04:33:19] PROBLEM Disk Space is now: CRITICAL on worker1 i-00000208 output: CHECK_NRPE: Socket timeout after 10 seconds. [04:33:19] PROBLEM Free ram is now: CRITICAL on worker1 i-00000208 output: CHECK_NRPE: Socket timeout after 10 seconds. [04:33:59] PROBLEM Current Users is now: CRITICAL on incubator-bot1 i-00000251 output: CHECK_NRPE: Socket timeout after 10 seconds. [04:33:59] PROBLEM dpkg-check is now: CRITICAL on incubator-bot1 i-00000251 output: CHECK_NRPE: Socket timeout after 10 seconds. [04:33:59] PROBLEM Total Processes is now: CRITICAL on incubator-bot1 i-00000251 output: CHECK_NRPE: Socket timeout after 10 seconds. [04:34:04] PROBLEM Disk Space is now: CRITICAL on rds i-00000207 output: CHECK_NRPE: Socket timeout after 10 seconds. [04:34:04] PROBLEM Free ram is now: CRITICAL on rds i-00000207 output: CHECK_NRPE: Socket timeout after 10 seconds. [04:34:04] PROBLEM Total Processes is now: CRITICAL on rds i-00000207 output: CHECK_NRPE: Socket timeout after 10 seconds. [04:35:11] PROBLEM Puppet freshness is now: CRITICAL on mailman-01 i-00000235 output: Puppet has not run in last 20 hours [04:36:56] RECOVERY Current Users is now: OK on reportcard2 i-000001ea output: USERS OK - 0 users currently logged in [04:36:56] RECOVERY Disk Space is now: OK on reportcard2 i-000001ea output: DISK OK [04:36:56] RECOVERY Free ram is now: OK on nova-daas-1 i-000000e7 output: OK: 94% free memory [04:37:26] RECOVERY Current Load is now: OK on maps-tilemill1 i-00000294 output: OK - load average: 0.37, 3.05, 3.54 [04:37:27] RECOVERY Total Processes is now: OK on maps-tilemill1 i-00000294 output: PROCS OK: 108 processes [04:37:32] RECOVERY Free ram is now: OK on maps-tilemill1 i-00000294 output: OK: 86% free memory [04:37:32] RECOVERY dpkg-check is now: OK on maps-tilemill1 i-00000294 output: All packages OK [04:37:49] RECOVERY Total Processes is now: OK on worker1 i-00000208 output: PROCS OK: 88 processes [04:37:54] RECOVERY Disk Space is now: OK on worker1 i-00000208 output: DISK OK [04:37:54] RECOVERY Free ram is now: OK on worker1 i-00000208 output: OK: 91% free memory [04:38:44] RECOVERY Current Users is now: OK on incubator-bot1 i-00000251 output: USERS OK - 0 users currently logged in [04:38:44] RECOVERY Total Processes is now: OK on incubator-bot1 i-00000251 output: PROCS OK: 125 processes [04:38:49] RECOVERY dpkg-check is now: OK on incubator-bot1 i-00000251 output: All packages OK [04:38:49] RECOVERY Disk Space is now: OK on rds i-00000207 output: DISK OK [04:38:49] RECOVERY Free ram is now: OK on rds i-00000207 output: OK: 92% free memory [04:38:49] RECOVERY Total Processes is now: OK on rds i-00000207 output: PROCS OK: 81 processes [05:03:43] PROBLEM Current Load is now: CRITICAL on mwreview-foo i-0000029a output: Connection refused by host [05:04:47] PROBLEM Current Users is now: CRITICAL on mwreview-foo i-0000029a output: Connection refused by host [05:05:28] PROBLEM Disk Space is now: CRITICAL on mwreview-foo i-0000029a output: Connection refused by host [05:05:57] PROBLEM Free ram is now: CRITICAL on mwreview-foo i-0000029a output: Connection refused by host [05:06:57] PROBLEM Total Processes is now: CRITICAL on mwreview-foo i-0000029a output: CHECK_NRPE: Error - Could not complete SSL handshake. [05:07:27] PROBLEM dpkg-check is now: CRITICAL on mwreview-foo i-0000029a output: CHECK_NRPE: Error - Could not complete SSL handshake. [06:13:57] RECOVERY Current Load is now: OK on bots-sql2 i-000000af output: OK - load average: 6.16, 4.39, 4.91 [06:21:57] PROBLEM Current Load is now: WARNING on bots-sql2 i-000000af output: WARNING - load average: 5.84, 5.52, 5.27 [06:33:09] PROBLEM Current Load is now: CRITICAL on nagios 127.0.0.1 output: CRITICAL - load average: 5.23, 6.75, 3.91 [06:36:53] PROBLEM Disk Space is now: CRITICAL on maps-tilemill1 i-00000294 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:39:14] PROBLEM Current Users is now: CRITICAL on maps-tilemill1 i-00000294 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:39:14] PROBLEM Current Load is now: CRITICAL on maps-tilemill1 i-00000294 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:39:14] PROBLEM Total Processes is now: CRITICAL on maps-tilemill1 i-00000294 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:39:19] PROBLEM Free ram is now: CRITICAL on maps-tilemill1 i-00000294 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:39:19] PROBLEM dpkg-check is now: CRITICAL on maps-tilemill1 i-00000294 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:39:19] PROBLEM Current Load is now: CRITICAL on bots-sql2 i-000000af output: CHECK_NRPE: Socket timeout after 10 seconds. [06:45:21] PROBLEM dpkg-check is now: CRITICAL on migration1 i-00000261 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:45:22] PROBLEM Current Users is now: CRITICAL on migration1 i-00000261 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:45:22] PROBLEM Disk Space is now: CRITICAL on mobile-testing i-00000271 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:45:22] PROBLEM Current Load is now: CRITICAL on mobile-testing i-00000271 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:45:22] PROBLEM Current Users is now: CRITICAL on mobile-testing i-00000271 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:45:22] PROBLEM Disk Space is now: CRITICAL on incubator-bot1 i-00000251 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:45:22] PROBLEM Current Load is now: CRITICAL on incubator-bot1 i-00000251 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:45:23] PROBLEM Current Users is now: CRITICAL on incubator-bot1 i-00000251 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:45:23] PROBLEM Free ram is now: CRITICAL on incubator-bot1 i-00000251 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:45:24] PROBLEM Total Processes is now: CRITICAL on incubator-bot1 i-00000251 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:45:28] PROBLEM Free ram is now: CRITICAL on mobile-testing i-00000271 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:45:28] PROBLEM Total Processes is now: CRITICAL on mobile-testing i-00000271 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:46:10] PROBLEM Current Load is now: WARNING on bots-cb i-0000009e output: WARNING - load average: 6.07, 10.76, 8.66 [06:47:08] PROBLEM Current Load is now: CRITICAL on migration1 i-00000261 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:47:20] PROBLEM Disk Space is now: CRITICAL on migration1 i-00000261 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:47:20] PROBLEM Free ram is now: CRITICAL on migration1 i-00000261 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:47:20] PROBLEM Total Processes is now: CRITICAL on migration1 i-00000261 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:47:25] PROBLEM dpkg-check is now: CRITICAL on incubator-bot1 i-00000251 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:49:06] PROBLEM Disk Space is now: CRITICAL on reportcard2 i-000001ea output: CHECK_NRPE: Socket timeout after 10 seconds. [06:49:06] PROBLEM Current Users is now: CRITICAL on reportcard2 i-000001ea output: CHECK_NRPE: Socket timeout after 10 seconds. [06:49:16] PROBLEM Current Load is now: WARNING on aggregator-test3 i-00000293 output: WARNING - load average: 3.93, 5.80, 6.06 [06:49:42] RECOVERY Disk Space is now: OK on incubator-bot1 i-00000251 output: DISK OK [06:50:10] PROBLEM Current Load is now: WARNING on incubator-bot1 i-00000251 output: WARNING - load average: 11.72, 11.86, 7.82 [06:50:10] RECOVERY Current Users is now: OK on incubator-bot1 i-00000251 output: USERS OK - 0 users currently logged in [06:50:11] RECOVERY Free ram is now: OK on incubator-bot1 i-00000251 output: OK: 69% free memory [06:50:11] RECOVERY Total Processes is now: OK on incubator-bot1 i-00000251 output: PROCS OK: 124 processes [06:50:45] PROBLEM Current Load is now: CRITICAL on rds i-00000207 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:50:45] PROBLEM Current Users is now: CRITICAL on rds i-00000207 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:50:45] PROBLEM Disk Space is now: CRITICAL on rds i-00000207 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:50:45] PROBLEM Free ram is now: CRITICAL on rds i-00000207 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:50:45] PROBLEM Total Processes is now: CRITICAL on rds i-00000207 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:50:50] PROBLEM Current Load is now: CRITICAL on reportcard2 i-000001ea output: CHECK_NRPE: Socket timeout after 10 seconds. [06:50:50] PROBLEM dpkg-check is now: CRITICAL on reportcard2 i-000001ea output: CHECK_NRPE: Socket timeout after 10 seconds. [06:50:50] PROBLEM Total Processes is now: CRITICAL on reportcard2 i-000001ea output: CHECK_NRPE: Socket timeout after 10 seconds. [06:52:07] PROBLEM Free ram is now: CRITICAL on reportcard2 i-000001ea output: CHECK_NRPE: Socket timeout after 10 seconds. [06:52:32] PROBLEM Current Load is now: CRITICAL on bots-cb i-0000009e output: CHECK_NRPE: Socket timeout after 10 seconds. [06:53:17] PROBLEM Total Processes is now: CRITICAL on incubator-bot2 i-00000252 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:53:22] PROBLEM Disk Space is now: CRITICAL on pybal-precise i-00000289 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:53:22] PROBLEM Current Users is now: CRITICAL on pybal-precise i-00000289 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:53:22] PROBLEM Free ram is now: CRITICAL on pybal-precise i-00000289 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:53:22] PROBLEM Current Load is now: CRITICAL on pybal-precise i-00000289 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:53:22] PROBLEM dpkg-check is now: CRITICAL on pybal-precise i-00000289 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:53:27] PROBLEM Total Processes is now: CRITICAL on pybal-precise i-00000289 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:53:52] PROBLEM Current Load is now: WARNING on incubator-bot2 i-00000252 output: WARNING - load average: 12.50, 11.00, 6.52 [06:54:17] PROBLEM dpkg-check is now: CRITICAL on deployment-thumbproxy i-0000026b output: CHECK_NRPE: Socket timeout after 10 seconds. [06:55:23] PROBLEM Current Load is now: WARNING on labs-nfs1 i-0000005d output: WARNING - load average: 11.60, 10.58, 7.76 [06:55:23] PROBLEM Current Load is now: WARNING on dumps-1 i-00000170 output: WARNING - load average: 8.13, 10.05, 6.05 [06:55:23] PROBLEM Current Load is now: WARNING on bots-apache1 i-000000b0 output: WARNING - load average: 6.92, 6.35, 5.86 [06:55:23] PROBLEM Total Processes is now: WARNING on nagios 127.0.0.1 output: PROCS WARNING: 265 processes [06:55:33] PROBLEM Current Load is now: CRITICAL on incubator-bot1 i-00000251 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:55:48] PROBLEM Free ram is now: CRITICAL on bots-cb i-0000009e output: CHECK_NRPE: Socket timeout after 10 seconds. [06:55:48] PROBLEM Total Processes is now: CRITICAL on bots-cb i-0000009e output: CHECK_NRPE: Socket timeout after 10 seconds. [06:55:54] PROBLEM SSH is now: CRITICAL on reportcard2 i-000001ea output: CRITICAL - Socket timeout after 10 seconds [06:55:54] PROBLEM Free ram is now: CRITICAL on labs-lvs1 i-00000057 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:55:54] PROBLEM dpkg-check is now: CRITICAL on bots-cb i-0000009e output: CHECK_NRPE: Socket timeout after 10 seconds. [06:55:54] PROBLEM dpkg-check is now: CRITICAL on labs-lvs1 i-00000057 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:55:54] PROBLEM Current Load is now: CRITICAL on test3 i-00000093 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:55:54] PROBLEM Disk Space is now: CRITICAL on test3 i-00000093 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:55:55] PROBLEM Current Users is now: CRITICAL on test3 i-00000093 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:55:55] PROBLEM Total Processes is now: CRITICAL on test3 i-00000093 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:55:58] PROBLEM Free ram is now: CRITICAL on bots-sql2 i-000000af output: CHECK_NRPE: Socket timeout after 10 seconds. [06:55:59] PROBLEM Free ram is now: CRITICAL on test3 i-00000093 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:55:59] PROBLEM Current Users is now: CRITICAL on labs-lvs1 i-00000057 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:55:59] PROBLEM Free ram is now: CRITICAL on utils-abogott i-00000131 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:55:59] PROBLEM Disk Space is now: CRITICAL on bots-cb i-0000009e output: CHECK_NRPE: Socket timeout after 10 seconds. [06:55:59] PROBLEM Current Users is now: CRITICAL on bots-cb i-0000009e output: CHECK_NRPE: Socket timeout after 10 seconds. [06:55:59] PROBLEM Total Processes is now: CRITICAL on bots-sql1 i-000000b5 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:00:50] PROBLEM Total Processes is now: CRITICAL on bots-sql2 i-000000af output: CHECK_NRPE: Socket timeout after 10 seconds. [07:00:55] PROBLEM Current Load is now: CRITICAL on labs-lvs1 i-00000057 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:00:55] PROBLEM dpkg-check is now: CRITICAL on bots-sql2 i-000000af output: CHECK_NRPE: Socket timeout after 10 seconds. [07:00:55] PROBLEM SSH is now: CRITICAL on bots-sql2 i-000000af output: CRITICAL - Socket timeout after 10 seconds [07:01:05] PROBLEM SSH is now: CRITICAL on bots-cb i-0000009e output: CRITICAL - Socket timeout after 10 seconds [07:01:05] PROBLEM Total Processes is now: CRITICAL on labs-lvs1 i-00000057 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:01:10] PROBLEM dpkg-check is now: CRITICAL on test3 i-00000093 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:01:10] PROBLEM Total Processes is now: CRITICAL on shop-analytics-main i-000001e6 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:01:15] PROBLEM HTTP is now: CRITICAL on resourceloader2-apache i-000001d7 output: CRITICAL - Socket timeout after 10 seconds [07:01:15] PROBLEM SSH is now: CRITICAL on rds i-00000207 output: CRITICAL - Socket timeout after 10 seconds [07:01:15] PROBLEM Disk Space is now: CRITICAL on labs-lvs1 i-00000057 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:04:53] PROBLEM Total Processes is now: CRITICAL on deployment-apache23 i-00000270 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:05:13] PROBLEM Current Load is now: CRITICAL on incubator-bot2 i-00000252 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:05:48] RECOVERY Current Load is now: OK on aggregator-test3 i-00000293 output: OK - load average: 4.80, 3.29, 4.02 [07:05:48] PROBLEM Current Load is now: WARNING on reportcard2 i-000001ea output: WARNING - load average: 16.64, 15.92, 11.42 [07:05:49] RECOVERY Total Processes is now: OK on reportcard2 i-000001ea output: PROCS OK: 93 processes [07:06:52] RECOVERY dpkg-check is now: OK on reportcard2 i-000001ea output: All packages OK [07:09:15] RECOVERY Disk Space is now: OK on maps-tilemill1 i-00000294 output: DISK OK [07:09:16] PROBLEM Current Load is now: WARNING on maps-tilemill1 i-00000294 output: WARNING - load average: 6.03, 8.71, 9.85 [07:09:16] RECOVERY Current Users is now: OK on maps-tilemill1 i-00000294 output: USERS OK - 0 users currently logged in [07:09:16] RECOVERY Total Processes is now: OK on maps-tilemill1 i-00000294 output: PROCS OK: 110 processes [07:09:21] RECOVERY Free ram is now: OK on maps-tilemill1 i-00000294 output: OK: 86% free memory [07:09:21] RECOVERY dpkg-check is now: OK on maps-tilemill1 i-00000294 output: All packages OK [07:09:21] RECOVERY Free ram is now: OK on labs-lvs1 i-00000057 output: OK: 90% free memory [07:09:21] RECOVERY Total Processes is now: OK on nagios 127.0.0.1 output: PROCS OK: 228 processes [07:09:21] RECOVERY SSH is now: OK on reportcard2 i-000001ea output: SSH OK - OpenSSH_5.8p1 Debian-7ubuntu1 (protocol 2.0) [07:09:21] RECOVERY dpkg-check is now: OK on labs-lvs1 i-00000057 output: All packages OK [07:09:21] RECOVERY Current Load is now: OK on test3 i-00000093 output: OK - load average: 0.82, 1.74, 2.03 [07:09:22] RECOVERY Disk Space is now: OK on test3 i-00000093 output: DISK OK [07:09:22] RECOVERY Current Users is now: OK on test3 i-00000093 output: USERS OK - 0 users currently logged in [07:09:23] RECOVERY Total Processes is now: OK on bots-cb i-0000009e output: PROCS OK: 196 processes [07:09:26] RECOVERY Free ram is now: OK on bots-cb i-0000009e output: OK: 76% free memory [07:09:26] RECOVERY dpkg-check is now: OK on bots-cb i-0000009e output: All packages OK [07:09:26] RECOVERY Current Users is now: OK on rds i-00000207 output: USERS OK - 0 users currently logged in [07:09:26] RECOVERY Disk Space is now: OK on rds i-00000207 output: DISK OK [07:09:26] PROBLEM Current Load is now: WARNING on rds i-00000207 output: WARNING - load average: 4.57, 7.29, 8.26 [07:09:26] RECOVERY Total Processes is now: OK on rds i-00000207 output: PROCS OK: 102 processes [07:09:31] RECOVERY Free ram is now: OK on rds i-00000207 output: OK: 92% free memory [07:09:41] RECOVERY Total Processes is now: OK on test3 i-00000093 output: PROCS OK: 75 processes [07:10:59] RECOVERY Free ram is now: OK on utils-abogott i-00000131 output: OK: 96% free memory [07:10:59] RECOVERY Free ram is now: OK on test3 i-00000093 output: OK: 53% free memory [07:10:59] RECOVERY Current Users is now: OK on bots-cb i-0000009e output: USERS OK - 0 users currently logged in [07:10:59] RECOVERY Disk Space is now: OK on bots-cb i-0000009e output: DISK OK [07:10:59] RECOVERY Total Processes is now: OK on bots-sql1 i-000000b5 output: PROCS OK: 84 processes [07:11:19] RECOVERY Total Processes is now: OK on shop-analytics-main i-000001e6 output: PROCS OK: 91 processes [07:11:24] RECOVERY HTTP is now: OK on resourceloader2-apache i-000001d7 output: HTTP OK: HTTP/1.1 200 OK - 453 bytes in 0.016 second response time [07:11:25] RECOVERY dpkg-check is now: OK on test3 i-00000093 output: All packages OK [07:11:25] RECOVERY SSH is now: OK on rds i-00000207 output: SSH OK - OpenSSH_5.8p1 Debian-7ubuntu1 (protocol 2.0) [07:11:25] PROBLEM Current Load is now: CRITICAL on precise-test i-00000231 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:11:25] PROBLEM dpkg-check is now: CRITICAL on mobile-testing i-00000271 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:11:25] PROBLEM Current Load is now: CRITICAL on worker1 i-00000208 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:11:25] PROBLEM Current Users is now: CRITICAL on worker1 i-00000208 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:11:26] PROBLEM Disk Space is now: CRITICAL on worker1 i-00000208 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:11:27] PROBLEM Free ram is now: CRITICAL on worker1 i-00000208 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:11:27] PROBLEM Total Processes is now: CRITICAL on worker1 i-00000208 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:11:30] PROBLEM Free ram is now: CRITICAL on precise-test i-00000231 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:11:30] PROBLEM Current Users is now: CRITICAL on precise-test i-00000231 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:11:30] PROBLEM Disk Space is now: CRITICAL on precise-test i-00000231 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:11:30] PROBLEM Total Processes is now: CRITICAL on precise-test i-00000231 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:11:35] PROBLEM dpkg-check is now: CRITICAL on precise-test i-00000231 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:13:42] RECOVERY Total Processes is now: OK on deployment-apache23 i-00000270 output: PROCS OK: 137 processes [07:13:59] PROBLEM Current Load is now: WARNING on bots-cb i-0000009e output: WARNING - load average: 9.05, 10.26, 18.23 [07:14:04] RECOVERY dpkg-check is now: OK on incubator-bot1 i-00000251 output: All packages OK [07:17:47] PROBLEM Current Load is now: WARNING on swift-be2 i-000001c8 output: WARNING - load average: 6.20, 7.21, 7.70 [07:17:47] PROBLEM Current Load is now: WARNING on deployment-apache23 i-00000270 output: WARNING - load average: 5.17, 5.78, 6.48 [07:17:47] PROBLEM Current Load is now: WARNING on swift-be4 i-000001ca output: WARNING - load average: 6.37, 7.42, 7.34 [07:17:57] RECOVERY Current Users is now: OK on migration1 i-00000261 output: USERS OK - 0 users currently logged in [07:17:57] RECOVERY dpkg-check is now: OK on migration1 i-00000261 output: All packages OK [07:17:57] PROBLEM Current Load is now: CRITICAL on rds i-00000207 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:18:02] PROBLEM Current Load is now: CRITICAL on reportcard2 i-000001ea output: CHECK_NRPE: Socket timeout after 10 seconds. [07:18:15] PROBLEM Current Load is now: CRITICAL on maps-tilemill1 i-00000294 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:21:59] PROBLEM Current Load is now: CRITICAL on dumps-1 i-00000170 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:22:41] RECOVERY Current Users is now: OK on labs-lvs1 i-00000057 output: USERS OK - 0 users currently logged in [07:22:41] RECOVERY Current Load is now: OK on labs-lvs1 i-00000057 output: OK - load average: 0.48, 1.98, 4.00 [07:22:41] PROBLEM Current Load is now: WARNING on precise-test i-00000231 output: WARNING - load average: 5.86, 7.14, 8.05 [07:22:41] RECOVERY dpkg-check is now: OK on mobile-testing i-00000271 output: All packages OK [07:22:41] RECOVERY Current Users is now: OK on precise-test i-00000231 output: USERS OK - 0 users currently logged in [07:22:41] RECOVERY Free ram is now: OK on precise-test i-00000231 output: OK: 83% free memory [07:22:42] RECOVERY Disk Space is now: OK on precise-test i-00000231 output: DISK OK [07:22:42] RECOVERY Total Processes is now: OK on precise-test i-00000231 output: PROCS OK: 88 processes [07:22:55] RECOVERY Total Processes is now: OK on labs-lvs1 i-00000057 output: PROCS OK: 79 processes [07:23:15] RECOVERY Disk Space is now: OK on labs-lvs1 i-00000057 output: DISK OK [07:23:15] RECOVERY SSH is now: OK on bots-cb i-0000009e output: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [07:23:15] RECOVERY dpkg-check is now: OK on precise-test i-00000231 output: All packages OK [07:23:15] RECOVERY Free ram is now: OK on mobile-testing i-00000271 output: OK: 69% free memory [07:23:15] PROBLEM Current Load is now: WARNING on migration1 i-00000261 output: WARNING - load average: 0.54, 3.37, 6.42 [07:23:15] RECOVERY Total Processes is now: OK on mobile-testing i-00000271 output: PROCS OK: 238 processes [07:23:40] RECOVERY Disk Space is now: OK on migration1 i-00000261 output: DISK OK [07:24:09] RECOVERY Free ram is now: OK on migration1 i-00000261 output: OK: 79% free memory [07:24:10] RECOVERY Total Processes is now: OK on migration1 i-00000261 output: PROCS OK: 86 processes [07:24:55] RECOVERY Total Processes is now: OK on incubator-bot2 i-00000252 output: PROCS OK: 123 processes [07:25:27] PROBLEM Current Load is now: WARNING on incubator-bot2 i-00000252 output: WARNING - load average: 7.35, 7.69, 7.71 [07:25:27] RECOVERY dpkg-check is now: OK on deployment-thumbproxy i-0000026b output: All packages OK [07:25:27] PROBLEM Current Load is now: WARNING on pybal-precise i-00000289 output: WARNING - load average: 3.19, 4.28, 6.06 [07:25:27] RECOVERY Disk Space is now: OK on pybal-precise i-00000289 output: DISK OK [07:25:27] RECOVERY Free ram is now: OK on pybal-precise i-00000289 output: OK: 80% free memory [07:25:27] RECOVERY Total Processes is now: OK on pybal-precise i-00000289 output: PROCS OK: 92 processes [07:25:37] RECOVERY Current Users is now: OK on pybal-precise i-00000289 output: USERS OK - 0 users currently logged in [07:27:06] RECOVERY Disk Space is now: OK on mobile-testing i-00000271 output: DISK OK [07:27:07] RECOVERY Current Users is now: OK on mobile-testing i-00000271 output: USERS OK - 0 users currently logged in [07:27:07] RECOVERY Current Load is now: OK on deployment-apache23 i-00000270 output: OK - load average: 0.96, 1.56, 3.87 [07:27:07] RECOVERY Current Load is now: OK on swift-be4 i-000001ca output: OK - load average: 2.71, 2.61, 4.77 [07:27:07] RECOVERY Current Load is now: OK on dumps-1 i-00000170 output: OK - load average: 1.91, 3.16, 4.89 [07:29:31] PROBLEM Current Load is now: WARNING on upload-wizard i-0000021c output: WARNING - load average: 5.46, 6.88, 6.15 [07:29:41] RECOVERY Current Load is now: OK on migration1 i-00000261 output: OK - load average: 0.28, 1.39, 4.71 [07:29:47] PROBLEM dpkg-check is now: CRITICAL on incubator-bot2 i-00000252 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:29:54] PROBLEM Current Load is now: CRITICAL on precise-test i-00000231 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:30:26] RECOVERY dpkg-check is now: OK on pybal-precise i-00000289 output: All packages OK [07:31:44] RECOVERY Current Load is now: OK on swift-be2 i-000001c8 output: OK - load average: 0.35, 2.31, 4.96 [07:32:39] RECOVERY dpkg-check is now: OK on incubator-bot2 i-00000252 output: All packages OK [07:32:39] RECOVERY Free ram is now: OK on bots-sql2 i-000000af output: OK: 76% free memory [07:32:39] RECOVERY Total Processes is now: OK on bots-sql2 i-000000af output: PROCS OK: 93 processes [07:32:44] RECOVERY dpkg-check is now: OK on bots-sql2 i-000000af output: All packages OK [07:32:45] RECOVERY SSH is now: OK on bots-sql2 i-000000af output: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [07:33:24] PROBLEM Current Load is now: WARNING on ganglia-test2 i-00000250 output: WARNING - load average: 1.53, 5.23, 6.67 [07:35:15] RECOVERY Current Load is now: OK on pybal-precise i-00000289 output: OK - load average: 0.11, 1.78, 3.90 [07:37:36] RECOVERY Current Load is now: OK on upload-wizard i-0000021c output: OK - load average: 0.08, 1.56, 3.81 [07:37:36] RECOVERY Current Load is now: OK on worker1 i-00000208 output: OK - load average: 0.10, 2.25, 4.24 [07:37:36] RECOVERY Current Users is now: OK on worker1 i-00000208 output: USERS OK - 0 users currently logged in [07:37:36] RECOVERY Disk Space is now: OK on worker1 i-00000208 output: DISK OK [07:37:36] RECOVERY Free ram is now: OK on worker1 i-00000208 output: OK: 91% free memory [07:37:36] RECOVERY Total Processes is now: OK on worker1 i-00000208 output: PROCS OK: 90 processes [07:37:41] RECOVERY Free ram is now: OK on reportcard2 i-000001ea output: OK: 85% free memory [07:37:42] RECOVERY Current Users is now: OK on reportcard2 i-000001ea output: USERS OK - 0 users currently logged in [07:37:42] RECOVERY Disk Space is now: OK on reportcard2 i-000001ea output: DISK OK [07:41:42] PROBLEM Current Load is now: WARNING on mobile-testing i-00000271 output: WARNING - load average: 0.48, 5.96, 16.72 [07:43:22] RECOVERY Current Load is now: OK on ganglia-test2 i-00000250 output: OK - load average: 0.34, 1.25, 3.80 [07:46:52] RECOVERY Current Load is now: OK on bots-apache1 i-000000b0 output: OK - load average: 3.17, 3.28, 4.49 [07:52:38] PROBLEM Current Load is now: WARNING on nagios 127.0.0.1 output: WARNING - load average: 5.80, 2.25, 3.58 [07:57:36] RECOVERY Current Load is now: OK on nagios 127.0.0.1 output: OK - load average: 2.27, 1.71, 2.93 [08:06:57] RECOVERY Current Load is now: OK on mobile-testing i-00000271 output: OK - load average: 1.28, 1.11, 4.01 [08:14:56] !log deployment-prep hashar: raising mysql max_connect_errors (bug 37173) [08:16:07] !log deployment-prep hashar: restarting mysql [08:33:01] !log deployment-prep hashar: raised max_connect_errors to 10000 and running FLUSH HOSTS; for bug 37173 [08:37:09] 05/29/2012 - 08:37:09 - Updating keys for ashley at /export/home/hackathon/ashley [08:37:16] 05/29/2012 - 08:37:16 - Updating keys for ashley at /export/home/bastion/ashley [09:07:42] !log deployment-prep hashar: Banned all Amazon Elastic Cloud IP ranges at squid level. Crawlers / spiders there are hammering the beta cluster. [09:10:39] Probably a good idea to block tor etc :P [09:12:08] yup Tor will probably be the next [09:12:19] could do it if you find out a list of Tor output nodes [09:12:28] coffee break [09:18:46] PROBLEM Free ram is now: WARNING on bots-3 i-000000e5 output: Warning: 17% free memory [09:23:46] RECOVERY Free ram is now: OK on bots-3 i-000000e5 output: OK: 21% free memory [09:53:21] PROBLEM Puppet freshness is now: CRITICAL on nova-ldap1 i-000000df output: Puppet has not run in last 20 hours [10:41:25] PROBLEM Free ram is now: WARNING on bots-3 i-000000e5 output: Warning: 19% free memory [10:57:25] mutante: can you arrange to get your first name showing up on your hackaton badge ? [10:57:41] mutante: cause I still cant remember if it is Daniel or David ;-D [11:02:57] hashar: only 48 hours until the first hackatonian code shall be written [11:05:56] yeah [11:06:02] still have to write a plan of action [12:16:11] PROBLEM Current Load is now: WARNING on mobile-testing i-00000271 output: WARNING - load average: 11.93, 10.52, 5.71 [12:21:44] !log bots restarting labs-morebots on bots-2 [12:21:45] bots is not a valid project. [12:21:46] Logged the message, Master [12:21:54] !log bot restarting labs-morebots on bots-2 [12:21:54] bot is not a valid project. [12:21:54] bot is not a valid project. [12:22:00] eh... [12:22:07] lol wut [12:22:13] I restarted it first before [12:22:28] * Hydriz pushes this for mutante to fix [12:23:37] eh, didnt you just ask me to do that? [12:23:40] ok [12:24:03] lol I was asking for the logging :P [12:24:08] but nevermind [12:24:17] it doesn't seem working? [12:24:25] !log foo [12:24:25] Message missing. Nothing logged. [12:24:32] !log foo foo [12:24:32] foo is not a valid project. [12:24:38] !log wikistats foo [12:24:39] Logged the message, Master [12:24:48] looks normal to me [12:24:55] hmm, seems like that was a conflict earlier then :P [12:25:02] yeah, i guess so [12:33:56] PROBLEM Current Load is now: CRITICAL on localpuppet2 i-0000029b output: Connection refused by host [12:34:36] PROBLEM Current Load is now: WARNING on bots-sql2 i-000000af output: WARNING - load average: 6.38, 5.74, 5.74 [12:34:46] PROBLEM Current Users is now: CRITICAL on localpuppet2 i-0000029b output: Connection refused by host [12:35:21] PROBLEM Disk Space is now: CRITICAL on localpuppet2 i-0000029b output: Connection refused by host [12:35:46] PROBLEM Free ram is now: CRITICAL on localpuppet2 i-0000029b output: Connection refused by host [12:37:16] PROBLEM Total Processes is now: CRITICAL on localpuppet2 i-0000029b output: Connection refused by host [12:37:46] PROBLEM dpkg-check is now: CRITICAL on localpuppet2 i-0000029b output: Connection refused by host [12:37:57] RECOVERY Current Users is now: OK on mwreview-foo i-0000029a output: USERS OK - 1 users currently logged in [12:37:57] RECOVERY Disk Space is now: OK on mwreview-foo i-0000029a output: DISK OK [12:37:57] RECOVERY Free ram is now: OK on mwreview-foo i-0000029a output: OK: 92% free memory [12:38:46] RECOVERY Total Processes is now: OK on mwreview-foo i-0000029a output: PROCS OK: 84 processes [12:38:51] RECOVERY dpkg-check is now: OK on mwreview-foo i-0000029a output: All packages OK [12:41:09] RECOVERY Current Load is now: OK on mwreview-foo i-0000029a output: OK - load average: 1.60, 1.23, 0.68 [12:41:57] PROBLEM Free ram is now: WARNING on bots-2 i-0000009c output: Warning: 19% free memory [12:48:52] RECOVERY Current Load is now: OK on localpuppet2 i-0000029b output: OK - load average: 1.30, 1.02, 1.07 [12:50:20] RECOVERY Current Users is now: OK on localpuppet2 i-0000029b output: USERS OK - 1 users currently logged in [12:50:40] RECOVERY Free ram is now: OK on localpuppet2 i-0000029b output: OK: 87% free memory [12:51:43] RECOVERY Disk Space is now: OK on localpuppet2 i-0000029b output: DISK OK [12:52:13] RECOVERY Total Processes is now: OK on localpuppet2 i-0000029b output: PROCS OK: 84 processes [12:52:43] RECOVERY dpkg-check is now: OK on localpuppet2 i-0000029b output: All packages OK [13:06:43] !log bots petrb: patching wm-bot [13:06:44] Logged the message, Master [13:07:32] thank you, its flooding irc.wikimedia.org [13:07:35] :) [13:15:11] RECOVERY Current Load is now: OK on bots-sql2 i-000000af output: OK - load average: 3.34, 4.19, 4.96 [13:20:30] Hydriz: what [13:20:37] who is flooding irc [13:21:11] wm-bot on irc.wm.o [13:21:13] how [13:21:25] join / part [13:21:26] maybe [13:21:32] because I was restarting my test [13:21:35] like 40 times [13:21:36] its quitting and joining and quitting and joining... [13:21:36] yeah [13:21:41] that's ok [13:22:01] but at least I am using my own bot for some of the channels :) [13:22:13] you can remove the incubator.wikimedia entry actually [13:22:22] why... [13:22:27] it doesn't matter to bot [13:22:35] I will remove channels only when projects are closed [13:22:55] maybe some other channel is using that feed [13:23:14] okie [13:26:19] RECOVERY Current Load is now: OK on mobile-testing i-00000271 output: OK - load average: 3.50, 3.70, 4.99 [13:35:44] PROBLEM dpkg-check is now: CRITICAL on localpuppet2 i-0000029b output: DPKG CRITICAL dpkg reports broken packages [13:43:47] PROBLEM Puppet freshness is now: CRITICAL on nova-precise1 i-00000236 output: Puppet has not run in last 20 hours [13:48:04] 05/29/2012 - 13:48:04 - Updating keys for mah at /export/home/wikistats/mah [13:48:17] 05/29/2012 - 13:48:16 - Updating keys for mah at /export/home/bastion/mah [13:48:21] PROBLEM Puppet freshness is now: CRITICAL on nova-essex-test i-000001f9 output: Puppet has not run in last 20 hours [13:48:21] 05/29/2012 - 13:48:21 - Updating keys for mah at /export/home/deployment-prep/mah [13:53:35] PROBLEM Current Load is now: WARNING on bots-sql2 i-000000af output: WARNING - load average: 5.10, 5.20, 5.06 [13:54:03] @infobot-share-on [13:54:03] Channel was configured to share the db [13:54:15] @infobot-share-trust+ #wikimedia-operations [13:54:15]