[00:00:06] it should work, but I can't guarantee anything as I haven't got it running myself either yet [00:00:25] your bot I mean - my bots work fine on python [00:02:15] Thehelpfulone: how much Java do you know? [00:02:40] Good luck getting things working, Sven_Manguard ... IIRC there are other folks around who know Java, including in the #mediawiki channel. I think Beau_ there knows Java [00:02:52] not much sorry [00:03:11] I'm looking into this x-forwarding [00:03:14] I know none [00:03:14] petan: am I right in remembering that you know some Java? [00:03:25] Thehelpfulone: x-forwarding is a pretty useful trick, I hear from the people who do it! [00:03:39] petan|wk: ^ (re Java) [00:05:03] thehelpfulone@bots-3:~$ The program 'xeyes' is currently not installed. To run 'xeyes' please ask your administrator to install the package 'x11-apps' [00:05:23] do I need that Reedy? ^ [00:05:35] probably [00:06:00] might as well give it to me too... [00:06:24] Sven_Manguard: it will be installed on the server to be accessible by all [00:06:59] petan: petan|wk ^^ something for you to have a look at please: thehelpfulone@bots-3:~$ The program 'xeyes' is currently not installed. To run 'xeyes' please ask your administrator to install the package 'x11-apps [00:12:37] night, all [01:34:10] PROBLEM Puppet freshness is now: CRITICAL on hugglewiki i-000000aa output: Puppet has not run in last 20 hours [01:35:10] PROBLEM Puppet freshness is now: CRITICAL on wikistream-1 i-0000016e output: Puppet has not run in last 20 hours [01:43:50] PROBLEM Free ram is now: WARNING on bots-2 i-0000009c output: Warning: 19% free memory [01:46:10] PROBLEM Puppet freshness is now: CRITICAL on migration1 i-00000261 output: Puppet has not run in last 20 hours [01:58:49] RECOVERY Free ram is now: OK on bots-2 i-0000009c output: OK: 20% free memory [02:00:14] PROBLEM Puppet freshness is now: CRITICAL on memcache-puppet i-00000153 output: Puppet has not run in last 20 hours [02:04:14] PROBLEM Puppet freshness is now: CRITICAL on jenkins2 i-00000102 output: Puppet has not run in last 20 hours [02:04:14] PROBLEM Puppet freshness is now: CRITICAL on publicdata-administration i-0000019e output: Puppet has not run in last 20 hours [02:04:14] PROBLEM Puppet freshness is now: CRITICAL on resourceloader2-apache i-000001d7 output: Puppet has not run in last 20 hours [02:05:14] PROBLEM Puppet freshness is now: CRITICAL on dumps-6 i-00000266 output: Puppet has not run in last 20 hours [02:05:14] PROBLEM Puppet freshness is now: CRITICAL on labs-nfs1 i-0000005d output: Puppet has not run in last 20 hours [02:05:14] PROBLEM Puppet freshness is now: CRITICAL on mailman-01 i-00000235 output: Puppet has not run in last 20 hours [02:05:14] PROBLEM Puppet freshness is now: CRITICAL on swift-be3 i-000001c9 output: Puppet has not run in last 20 hours [02:05:14] PROBLEM Puppet freshness is now: CRITICAL on wikidata-dev-3 i-00000225 output: Puppet has not run in last 20 hours [02:06:14] PROBLEM Puppet freshness is now: CRITICAL on wep i-000000c2 output: Puppet has not run in last 20 hours [02:07:24] PROBLEM Puppet freshness is now: CRITICAL on deployment-thumbproxy i-0000026b output: Puppet has not run in last 20 hours [02:07:24] PROBLEM Puppet freshness is now: CRITICAL on gerrit i-000000ff output: Puppet has not run in last 20 hours [02:07:24] PROBLEM Puppet freshness is now: CRITICAL on gluster-devstack i-00000274 output: Puppet has not run in last 20 hours [02:07:24] PROBLEM Puppet freshness is now: CRITICAL on shop-analytics-main i-000001e6 output: Puppet has not run in last 20 hours [02:07:24] PROBLEM Puppet freshness is now: CRITICAL on wikistats-01 i-00000042 output: Puppet has not run in last 20 hours [02:07:24] PROBLEM Puppet freshness is now: CRITICAL on worker1 i-00000208 output: Puppet has not run in last 20 hours [02:08:14] PROBLEM Puppet freshness is now: CRITICAL on bots-apache1 i-000000b0 output: Puppet has not run in last 20 hours [02:08:14] PROBLEM Puppet freshness is now: CRITICAL on deployment-web3 i-00000219 output: Puppet has not run in last 20 hours [02:08:14] PROBLEM Puppet freshness is now: CRITICAL on dumps-2 i-00000257 output: Puppet has not run in last 20 hours [02:08:14] PROBLEM Puppet freshness is now: CRITICAL on precise-test i-00000231 output: Puppet has not run in last 20 hours [02:09:14] PROBLEM Puppet freshness is now: CRITICAL on swift-be2 i-000001c8 output: Puppet has not run in last 20 hours [02:10:14] PROBLEM Puppet freshness is now: CRITICAL on building i-0000014d output: Puppet has not run in last 20 hours [02:10:14] PROBLEM Puppet freshness is now: CRITICAL on deployment-backup i-000000f8 output: Puppet has not run in last 20 hours [02:12:14] PROBLEM Puppet freshness is now: CRITICAL on aggregator-test2 i-0000024e output: Puppet has not run in last 20 hours [02:12:14] PROBLEM Puppet freshness is now: CRITICAL on reportcard2 i-000001ea output: Puppet has not run in last 20 hours [02:13:14] PROBLEM Puppet freshness is now: CRITICAL on deployment-apache22 i-0000026f output: Puppet has not run in last 20 hours [02:13:14] PROBLEM Puppet freshness is now: CRITICAL on dumps-1 i-00000170 output: Puppet has not run in last 20 hours [02:13:14] PROBLEM Puppet freshness is now: CRITICAL on webserver-lcarr i-00000134 output: Puppet has not run in last 20 hours [02:15:14] PROBLEM Puppet freshness is now: CRITICAL on bots-4 i-000000e8 output: Puppet has not run in last 20 hours [02:15:14] PROBLEM Puppet freshness is now: CRITICAL on bots-cb i-0000009e output: Puppet has not run in last 20 hours [02:15:14] PROBLEM Puppet freshness is now: CRITICAL on ganglia-collector i-000000b7 output: Puppet has not run in last 20 hours [02:15:14] PROBLEM Puppet freshness is now: CRITICAL on hugglewa-db i-00000188 output: Puppet has not run in last 20 hours [02:15:14] PROBLEM Puppet freshness is now: CRITICAL on swift-be4 i-000001ca output: Puppet has not run in last 20 hours [02:15:14] PROBLEM Puppet freshness is now: CRITICAL on wikisource-web i-000000fe output: Puppet has not run in last 20 hours [02:16:14] PROBLEM Puppet freshness is now: CRITICAL on deployment-apache20 i-0000026c output: Puppet has not run in last 20 hours [02:16:14] PROBLEM Puppet freshness is now: CRITICAL on deployment-transcoding i-00000105 output: Puppet has not run in last 20 hours [02:16:14] PROBLEM Puppet freshness is now: CRITICAL on incubator-bot1 i-00000251 output: Puppet has not run in last 20 hours [02:17:14] PROBLEM Puppet freshness is now: CRITICAL on deployment-cache-upload i-00000263 output: Puppet has not run in last 20 hours [02:17:14] PROBLEM Puppet freshness is now: CRITICAL on deployment-dbdump i-000000d2 output: Puppet has not run in last 20 hours [02:17:14] PROBLEM Puppet freshness is now: CRITICAL on php5builds i-00000192 output: Puppet has not run in last 20 hours [02:18:14] PROBLEM Puppet freshness is now: CRITICAL on deployment-feed i-00000118 output: Puppet has not run in last 20 hours [02:18:14] PROBLEM Puppet freshness is now: CRITICAL on incubator-bot2 i-00000252 output: Puppet has not run in last 20 hours [02:18:14] PROBLEM Puppet freshness is now: CRITICAL on robh2 i-000001a2 output: Puppet has not run in last 20 hours [02:18:14] PROBLEM Puppet freshness is now: CRITICAL on wikidata-dev-1 i-0000020c output: Puppet has not run in last 20 hours [02:19:14] PROBLEM Puppet freshness is now: CRITICAL on bots-sql1 i-000000b5 output: Puppet has not run in last 20 hours [02:19:14] PROBLEM Puppet freshness is now: CRITICAL on cn-wiki-db-lucid i-00000241 output: Puppet has not run in last 20 hours [02:19:14] PROBLEM Puppet freshness is now: CRITICAL on firstinstance i-0000013e output: Puppet has not run in last 20 hours [02:19:14] PROBLEM Puppet freshness is now: CRITICAL on kripke i-00000268 output: Puppet has not run in last 20 hours [02:19:14] PROBLEM Puppet freshness is now: CRITICAL on localpuppet1 i-0000020b output: Puppet has not run in last 20 hours [02:20:14] PROBLEM Puppet freshness is now: CRITICAL on bots-sql2 i-000000af output: Puppet has not run in last 20 hours [02:20:14] PROBLEM Puppet freshness is now: CRITICAL on mobile-testing i-00000271 output: Puppet has not run in last 20 hours [02:20:14] PROBLEM Puppet freshness is now: CRITICAL on swift-fe1 i-000001d2 output: Puppet has not run in last 20 hours [02:22:14] PROBLEM Puppet freshness is now: CRITICAL on ganglia-test2 i-00000250 output: Puppet has not run in last 20 hours [02:22:14] PROBLEM Puppet freshness is now: CRITICAL on nginx-dev1 i-000000f0 output: Puppet has not run in last 20 hours [02:22:14] PROBLEM Puppet freshness is now: CRITICAL on secondinstance i-0000015b output: Puppet has not run in last 20 hours [02:22:14] PROBLEM Puppet freshness is now: CRITICAL on test2 i-0000013c output: Puppet has not run in last 20 hours [02:23:14] PROBLEM Puppet freshness is now: CRITICAL on deployment-apache23 i-00000270 output: Puppet has not run in last 20 hours [02:24:14] PROBLEM Puppet freshness is now: CRITICAL on deployment-webs1 i-0000012a output: Puppet has not run in last 20 hours [02:25:14] PROBLEM Puppet freshness is now: CRITICAL on bastion1 i-000000ba output: Puppet has not run in last 20 hours [02:26:14] PROBLEM Puppet freshness is now: CRITICAL on venus i-000000ea output: Puppet has not run in last 20 hours [02:27:22] PROBLEM Puppet freshness is now: CRITICAL on demo-deployment1 i-00000276 output: Puppet has not run in last 20 hours [02:30:14] PROBLEM Puppet freshness is now: CRITICAL on deployment-cache-bits i-00000264 output: Puppet has not run in last 20 hours [02:30:14] PROBLEM Puppet freshness is now: CRITICAL on feeds i-000000fa output: Puppet has not run in last 20 hours [02:30:14] PROBLEM Puppet freshness is now: CRITICAL on fundraising-civicrm i-00000169 output: Puppet has not run in last 20 hours [02:31:14] PROBLEM Puppet freshness is now: CRITICAL on deployment-nfs-memc i-000000d7 output: Puppet has not run in last 20 hours [02:31:14] PROBLEM Puppet freshness is now: CRITICAL on grail i-0000021e output: Puppet has not run in last 20 hours [02:31:14] PROBLEM Puppet freshness is now: CRITICAL on labs-relay i-00000103 output: Puppet has not run in last 20 hours [02:31:14] PROBLEM Puppet freshness is now: CRITICAL on nova-production1 i-0000007b output: Puppet has not run in last 20 hours [02:31:14] PROBLEM Puppet freshness is now: CRITICAL on udp-filter i-000001df output: Puppet has not run in last 20 hours [02:32:14] PROBLEM Puppet freshness is now: CRITICAL on bastion-restricted1 i-0000019b output: Puppet has not run in last 20 hours [02:32:14] PROBLEM Puppet freshness is now: CRITICAL on bots-2 i-0000009c output: Puppet has not run in last 20 hours [02:32:14] PROBLEM Puppet freshness is now: CRITICAL on demo-mysql1 i-00000256 output: Puppet has not run in last 20 hours [02:32:14] PROBLEM Puppet freshness is now: CRITICAL on demo-web1 i-00000255 output: Puppet has not run in last 20 hours [02:32:14] PROBLEM Puppet freshness is now: CRITICAL on deployment-imagescaler01 i-0000025a output: Puppet has not run in last 20 hours [02:32:14] PROBLEM Puppet freshness is now: CRITICAL on deployment-mc i-0000021b output: Puppet has not run in last 20 hours [02:32:15] PROBLEM Puppet freshness is now: CRITICAL on ee-prototype i-0000013d output: Puppet has not run in last 20 hours [02:32:15] PROBLEM Puppet freshness is now: CRITICAL on fr-wiki-db-precise i-0000023e output: Puppet has not run in last 20 hours [02:32:16] PROBLEM Puppet freshness is now: CRITICAL on swift-aux1 i-0000024b output: Puppet has not run in last 20 hours [02:33:14] PROBLEM Puppet freshness is now: CRITICAL on build-precise1 i-00000273 output: Puppet has not run in last 20 hours [02:34:14] PROBLEM Puppet freshness is now: CRITICAL on fundraising-db i-0000015c output: Puppet has not run in last 20 hours [02:34:14] PROBLEM Puppet freshness is now: CRITICAL on rds i-00000207 output: Puppet has not run in last 20 hours [02:34:14] PROBLEM Puppet freshness is now: CRITICAL on upload-wizard i-0000021c output: Puppet has not run in last 20 hours [02:35:14] PROBLEM Puppet freshness is now: CRITICAL on asher1 i-0000003a output: Puppet has not run in last 20 hours [02:35:14] PROBLEM Puppet freshness is now: CRITICAL on bots-1 i-000000a9 output: Puppet has not run in last 20 hours [02:35:14] PROBLEM Puppet freshness is now: CRITICAL on dumps-7 i-00000267 output: Puppet has not run in last 20 hours [02:36:14] PROBLEM Puppet freshness is now: CRITICAL on deployment-web i-00000217 output: Puppet has not run in last 20 hours [02:36:14] PROBLEM Puppet freshness is now: CRITICAL on dumps-5 i-00000258 output: Puppet has not run in last 20 hours [02:36:14] PROBLEM Puppet freshness is now: CRITICAL on pageviews i-000000b2 output: Puppet has not run in last 20 hours [02:36:14] PROBLEM Puppet freshness is now: CRITICAL on pediapress-ocg1 i-00000233 output: Puppet has not run in last 20 hours [02:39:58] PROBLEM Puppet freshness is now: CRITICAL on aggregator-test i-0000024d output: Puppet has not run in last 20 hours [02:39:58] PROBLEM Puppet freshness is now: CRITICAL on bots-labs i-0000015e output: Puppet has not run in last 20 hours [02:39:58] PROBLEM Puppet freshness is now: CRITICAL on deployment-wmsearch i-000000e1 output: Puppet has not run in last 20 hours [02:39:58] PROBLEM Puppet freshness is now: CRITICAL on embed-sandbox i-000000d1 output: Puppet has not run in last 20 hours [02:39:58] PROBLEM Puppet freshness is now: CRITICAL on en-wiki-db-lucid i-0000023b output: Puppet has not run in last 20 hours [02:39:58] PROBLEM Puppet freshness is now: CRITICAL on opengrok-web i-000001e1 output: Puppet has not run in last 20 hours [02:39:59] PROBLEM Puppet freshness is now: CRITICAL on p-b i-000000ae output: Puppet has not run in last 20 hours [02:39:59] PROBLEM Puppet freshness is now: CRITICAL on testing-groupchange i-00000205 output: Puppet has not run in last 20 hours [02:40:00] PROBLEM Puppet freshness is now: CRITICAL on ve-nodejs i-00000245 output: Puppet has not run in last 20 hours [02:40:21] PROBLEM Puppet freshness is now: CRITICAL on catsort-pub i-000001cc output: Puppet has not run in last 20 hours [02:40:21] PROBLEM Puppet freshness is now: CRITICAL on incubator-common i-00000254 output: Puppet has not run in last 20 hours [02:40:21] PROBLEM Puppet freshness is now: CRITICAL on otrs-jgreen i-0000015a output: Puppet has not run in last 20 hours [02:40:21] PROBLEM Puppet freshness is now: CRITICAL on outreacheval i-0000012e output: Puppet has not run in last 20 hours [02:40:21] PROBLEM Puppet freshness is now: CRITICAL on patchtest i-000000f1 output: Puppet has not run in last 20 hours [02:40:21] PROBLEM Puppet freshness is now: CRITICAL on ubuntu1-pgehres i-000000fb output: Puppet has not run in last 20 hours [02:42:04] PROBLEM Free ram is now: WARNING on bots-2 i-0000009c output: Warning: 18% free memory [02:42:14] PROBLEM Puppet freshness is now: CRITICAL on bots-dev i-00000190 output: Puppet has not run in last 20 hours [02:42:14] PROBLEM Puppet freshness is now: CRITICAL on dev-solr i-00000152 output: Puppet has not run in last 20 hours [02:42:14] PROBLEM Puppet freshness is now: CRITICAL on en-wiki-db-precise i-0000023c output: Puppet has not run in last 20 hours [02:42:14] PROBLEM Puppet freshness is now: CRITICAL on maps-test2 i-00000253 output: Puppet has not run in last 20 hours [02:42:14] PROBLEM Puppet freshness is now: CRITICAL on nova-dev3 i-000000e9 output: Puppet has not run in last 20 hours [02:43:04] PROBLEM Current Load is now: WARNING on nagios 127.0.0.1 output: WARNING - load average: 1.56, 4.57, 2.61 [02:43:14] PROBLEM Puppet freshness is now: CRITICAL on log1 i-00000239 output: Puppet has not run in last 20 hours [02:43:14] PROBLEM Puppet freshness is now: CRITICAL on vumi i-000001e5 output: Puppet has not run in last 20 hours [02:44:14] PROBLEM Puppet freshness is now: CRITICAL on deployment-web4 i-00000214 output: Puppet has not run in last 20 hours [02:44:14] PROBLEM Puppet freshness is now: CRITICAL on ganglia-test3 i-0000025b output: Puppet has not run in last 20 hours [02:48:04] RECOVERY Current Load is now: OK on nagios 127.0.0.1 output: OK - load average: 0.46, 1.85, 1.95 [02:50:14] PROBLEM Puppet freshness is now: CRITICAL on aggregator1 i-0000010c output: Puppet has not run in last 20 hours [02:50:14] PROBLEM Puppet freshness is now: CRITICAL on bots-3 i-000000e5 output: Puppet has not run in last 20 hours [02:50:14] PROBLEM Puppet freshness is now: CRITICAL on deployment-web5 i-00000213 output: Puppet has not run in last 20 hours [02:50:14] PROBLEM Puppet freshness is now: CRITICAL on exim-test i-00000265 output: Puppet has not run in last 20 hours [02:50:14] PROBLEM Puppet freshness is now: CRITICAL on nova-essex-test i-000001f9 output: Puppet has not run in last 20 hours [02:50:14] PROBLEM Puppet freshness is now: CRITICAL on nova-ldap2 i-00000238 output: Puppet has not run in last 20 hours [02:50:15] PROBLEM Puppet freshness is now: CRITICAL on nova-precise1 i-00000236 output: Puppet has not run in last 20 hours [02:50:15] PROBLEM Puppet freshness is now: CRITICAL on scribunto i-0000022c output: Puppet has not run in last 20 hours [02:50:16] PROBLEM Puppet freshness is now: CRITICAL on swift-aux2 i-0000024c output: Puppet has not run in last 20 hours [02:50:16] PROBLEM Puppet freshness is now: CRITICAL on varnish i-000001ac output: Puppet has not run in last 20 hours [02:51:12] PROBLEM Puppet freshness is now: CRITICAL on deployment-apache21 i-0000026d output: Puppet has not run in last 20 hours [02:53:12] PROBLEM Puppet freshness is now: CRITICAL on deployment-sql i-000000d0 output: Puppet has not run in last 20 hours [02:53:12] PROBLEM Puppet freshness is now: CRITICAL on hugglewa-1 i-000001e0 output: Puppet has not run in last 20 hours [02:53:12] PROBLEM Puppet freshness is now: CRITICAL on master i-0000007a output: Puppet has not run in last 20 hours [02:53:12] PROBLEM Puppet freshness is now: CRITICAL on nginx-ffuqua-doom1-3 i-00000196 output: Puppet has not run in last 20 hours [02:53:12] PROBLEM Puppet freshness is now: CRITICAL on simplewikt i-00000149 output: Puppet has not run in last 20 hours [02:53:12] PROBLEM Puppet freshness is now: CRITICAL on vumi-gw1 i-0000008f output: Puppet has not run in last 20 hours [02:54:43] PROBLEM Puppet freshness is now: CRITICAL on labs-realserver i-00000104 output: Puppet has not run in last 20 hours [02:59:20] 05/17/2012 - 02:59:19 - Updating keys for laner at /export/home/deployment-prep/laner [02:59:24] PROBLEM Puppet freshness is now: CRITICAL on bob i-0000012d output: Puppet has not run in last 20 hours [02:59:24] PROBLEM Puppet freshness is now: CRITICAL on bots-nfs i-000000b1 output: Puppet has not run in last 20 hours [02:59:24] PROBLEM Puppet freshness is now: CRITICAL on bots-sql3 i-000000b4 output: Puppet has not run in last 20 hours [02:59:24] PROBLEM Puppet freshness is now: CRITICAL on deployment-squid i-000000dc output: Puppet has not run in last 20 hours [02:59:24] PROBLEM Puppet freshness is now: CRITICAL on dumps-3 i-0000017c output: Puppet has not run in last 20 hours [02:59:24] PROBLEM Puppet freshness is now: CRITICAL on dumps-8 i-0000026e output: Puppet has not run in last 20 hours [02:59:25] PROBLEM Puppet freshness is now: CRITICAL on labs-build1 i-0000006b output: Puppet has not run in last 20 hours [02:59:25] PROBLEM Puppet freshness is now: CRITICAL on pediapress-ocg2 i-00000234 output: Puppet has not run in last 20 hours [02:59:26] PROBLEM Puppet freshness is now: CRITICAL on pediapress-packager i-000001e4 output: Puppet has not run in last 20 hours [02:59:26] PROBLEM Puppet freshness is now: CRITICAL on translation-memory-1 i-0000013a output: Puppet has not run in last 20 hours [02:59:27] PROBLEM Puppet freshness is now: CRITICAL on vivek-puppet i-000000ca output: Puppet has not run in last 20 hours [03:00:15] PROBLEM Puppet freshness is now: CRITICAL on dumps-4 i-0000017d output: Puppet has not run in last 20 hours [03:00:15] PROBLEM Puppet freshness is now: CRITICAL on incubator-apache i-00000211 output: Puppet has not run in last 20 hours [03:11:57] PROBLEM Current Load is now: WARNING on nagios 127.0.0.1 output: WARNING - load average: 5.63, 5.47, 3.36 [03:21:31] 05/17/2012 - 03:21:31 - Updating keys for laner at /export/home/deployment-prep/laner [03:35:30] Ryan_Lane: how often do you update your keys? [03:35:35] or is this a bug [03:35:40] it's a bug [03:35:51] why only for you ? [03:35:55] not sure [03:36:02] need to figure it out [03:36:14] I'm wondering if it has something to do with OATHAuth [03:36:23] though I can't imagine it does [03:36:36] wgOSMRandomlyNagAboutUpdateKeysWhenUserJoinsIrc = array( 'laner' ); ? [03:36:48] heh [03:36:59] it was right before you joined [03:37:03] 05/17/2012 - 03:21:31 - Updating keys for laner at /export/home/deployment-prep/laner [03:37:15] and an hour before that as well [03:39:47] there must be something screwed up with my account in ldap [03:42:12] or with the script [03:42:18] I could be computing timestamps poorly [03:42:59] PROBLEM Free ram is now: WARNING on test-oneiric i-00000187 output: Warning: 14% free memory [03:43:14] still weird that it's only happening to me [03:43:20] 05/17/2012 - 03:43:19 - Updating keys for laner at /export/home/deployment-prep/laner [03:43:57] -_- [03:57:58] PROBLEM Free ram is now: CRITICAL on test-oneiric i-00000187 output: Critical: 5% free memory [04:00:29] PROBLEM Free ram is now: WARNING on utils-abogott i-00000131 output: Warning: 15% free memory [04:02:59] RECOVERY Free ram is now: OK on test-oneiric i-00000187 output: OK: 97% free memory [04:08:09] PROBLEM Free ram is now: WARNING on orgcharts-dev i-0000018f output: Warning: 14% free memory [04:10:29] PROBLEM Free ram is now: WARNING on nova-daas-1 i-000000e7 output: Warning: 15% free memory [04:20:39] PROBLEM Free ram is now: CRITICAL on utils-abogott i-00000131 output: Critical: 3% free memory [04:22:40] PROBLEM dpkg-check is now: CRITICAL on nova-daas-1 i-000000e7 output: CHECK_NRPE: Socket timeout after 10 seconds. [04:25:32] RECOVERY Free ram is now: OK on utils-abogott i-00000131 output: OK: 96% free memory [04:27:35] RECOVERY dpkg-check is now: OK on nova-daas-1 i-000000e7 output: All packages OK [04:28:15] PROBLEM Free ram is now: CRITICAL on orgcharts-dev i-0000018f output: Critical: 4% free memory [04:33:12] RECOVERY Free ram is now: OK on orgcharts-dev i-0000018f output: OK: 95% free memory [04:35:31] PROBLEM Free ram is now: CRITICAL on nova-daas-1 i-000000e7 output: Critical: 5% free memory [04:45:31] RECOVERY Free ram is now: OK on nova-daas-1 i-000000e7 output: OK: 94% free memory [04:49:16] PROBLEM Puppet freshness is now: CRITICAL on deployment-syslog i-00000269 output: Puppet has not run in last 20 hours [05:38:51] MySQL error: 1146: Table 'enwiki.pagetriage_page' doesn't exist (deployment-sql) [05:38:52] oh yeah [05:38:53] ;) [06:05:34] PROBLEM Disk Space is now: WARNING on nova-production1 i-0000007b output: DISK WARNING - free space: / 573 MB (5% inode=80%): [06:43:13] RECOVERY Disk Space is now: OK on nova-production1 i-0000007b output: DISK OK [06:47:21] !log labs died ;-D Veryyyy high I/O latency [06:47:21] labs is not a valid project. [06:47:24] !log [06:47:27] !help log [06:47:28] !documentation for labs !wm-bot for bot [06:47:38] !wm-bot [06:47:38] http://meta.wikimedia.org/wiki/WM-Bot [06:47:49] !help [06:47:49] !documentation for labs !wm-bot for bot [06:49:04] yeah, labs died [06:49:17] I did a :wq on a vi of mine and I'm waiting for like 10 minutes now [06:49:38] 06:49:28 up 16 days, 5:43, 2 users, load average: 482.89, 353.14, 203.70 [06:49:42] "yay" [06:52:34] PROBLEM Current Load is now: CRITICAL on nagios 127.0.0.1 output: CRITICAL - load average: 13.00, 12.54, 7.83 [06:53:35] PROBLEM Free ram is now: CRITICAL on bots-2 i-0000009c output: CHECK_NRPE: Socket timeout after 10 seconds. [06:54:00] PROBLEM dpkg-check is now: CRITICAL on incubator-bot1 i-00000251 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:54:20] PROBLEM Disk Space is now: CRITICAL on precise-test i-00000231 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:54:20] PROBLEM Total Processes is now: CRITICAL on precise-test i-00000231 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:54:25] PROBLEM dpkg-check is now: CRITICAL on precise-test i-00000231 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:54:25] PROBLEM Current Load is now: CRITICAL on incubator-bot1 i-00000251 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:54:26] PROBLEM Total Processes is now: CRITICAL on rds i-00000207 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:54:30] PROBLEM Free ram is now: CRITICAL on precise-test i-00000231 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:54:31] PROBLEM Free ram is now: CRITICAL on rds i-00000207 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:54:31] PROBLEM dpkg-check is now: CRITICAL on migration1 i-00000261 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:54:32] PROBLEM Current Load is now: CRITICAL on precise-test i-00000231 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:54:32] PROBLEM Current Users is now: CRITICAL on rds i-00000207 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:54:32] PROBLEM Free ram is now: CRITICAL on migration1 i-00000261 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:56:50] PROBLEM Current Load is now: CRITICAL on migration1 i-00000261 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:56:50] PROBLEM Current Users is now: CRITICAL on migration1 i-00000261 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:56:50] PROBLEM Disk Space is now: CRITICAL on migration1 i-00000261 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:56:50] PROBLEM Total Processes is now: CRITICAL on migration1 i-00000261 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:56:55] PROBLEM Disk Space is now: CRITICAL on rds i-00000207 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:57:23] PROBLEM Current Load is now: WARNING on rds i-00000207 output: WARNING - load average: 4.77, 7.40, 5.26 [06:57:24] PROBLEM Disk Space is now: CRITICAL on labs-lvs1 i-00000057 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:57:24] PROBLEM Disk Space is now: CRITICAL on incubator-bot1 i-00000251 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:57:24] PROBLEM Free ram is now: CRITICAL on incubator-bot1 i-00000251 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:57:24] PROBLEM Current Load is now: CRITICAL on labs-lvs1 i-00000057 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:57:24] PROBLEM Total Processes is now: CRITICAL on labs-lvs1 i-00000057 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:57:34] PROBLEM Free ram is now: CRITICAL on labs-lvs1 i-00000057 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:57:34] PROBLEM dpkg-check is now: CRITICAL on labs-lvs1 i-00000057 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:57:34] PROBLEM Current Users is now: CRITICAL on labs-lvs1 i-00000057 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:58:15] PROBLEM Free ram is now: WARNING on bots-2 i-0000009c output: Warning: 17% free memory [06:58:45] PROBLEM Current Users is now: CRITICAL on incubator-bot1 i-00000251 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:59:10] RECOVERY Disk Space is now: OK on precise-test i-00000231 output: DISK OK [06:59:10] RECOVERY Total Processes is now: OK on precise-test i-00000231 output: PROCS OK: 82 processes [06:59:15] PROBLEM Current Load is now: WARNING on incubator-bot1 i-00000251 output: WARNING - load average: 7.32, 8.45, 6.45 [06:59:15] RECOVERY dpkg-check is now: OK on precise-test i-00000231 output: All packages OK [06:59:15] RECOVERY Free ram is now: OK on precise-test i-00000231 output: OK: 86% free memory [06:59:15] RECOVERY Current Load is now: OK on precise-test i-00000231 output: OK - load average: 3.57, 5.10, 4.19 [07:01:44] PROBLEM Current Load is now: WARNING on mobile-testing i-00000271 output: WARNING - load average: 9.16, 8.23, 5.91 [07:01:59] RECOVERY Disk Space is now: OK on labs-lvs1 i-00000057 output: DISK OK [07:01:59] RECOVERY Free ram is now: OK on incubator-bot1 i-00000251 output: OK: 69% free memory [07:01:59] RECOVERY Current Load is now: OK on labs-lvs1 i-00000057 output: OK - load average: 2.23, 4.23, 3.56 [07:01:59] RECOVERY Total Processes is now: OK on labs-lvs1 i-00000057 output: PROCS OK: 78 processes [07:02:04] RECOVERY Disk Space is now: OK on incubator-bot1 i-00000251 output: DISK OK [07:02:04] RECOVERY Free ram is now: OK on labs-lvs1 i-00000057 output: OK: 91% free memory [07:02:05] RECOVERY Current Users is now: OK on labs-lvs1 i-00000057 output: USERS OK - 0 users currently logged in [07:02:05] RECOVERY dpkg-check is now: OK on labs-lvs1 i-00000057 output: All packages OK [07:02:09] PROBLEM Current Load is now: CRITICAL on rds i-00000207 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:07:33] RECOVERY dpkg-check is now: OK on migration1 i-00000261 output: All packages OK [07:07:33] RECOVERY Free ram is now: OK on migration1 i-00000261 output: OK: 84% free memory [07:07:33] RECOVERY Total Processes is now: OK on rds i-00000207 output: PROCS OK: 76 processes [07:07:38] RECOVERY Current Users is now: OK on rds i-00000207 output: USERS OK - 0 users currently logged in [07:07:38] RECOVERY Free ram is now: OK on rds i-00000207 output: OK: 94% free memory [07:07:43] PROBLEM Current Load is now: CRITICAL on incubator-bot1 i-00000251 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:08:30] PROBLEM HTTP is now: CRITICAL on deployment-web3 i-00000219 output: CRITICAL - Socket timeout after 10 seconds [07:08:31] RECOVERY Disk Space is now: OK on rds i-00000207 output: DISK OK [07:08:31] PROBLEM Current Load is now: WARNING on rds i-00000207 output: WARNING - load average: 0.88, 4.66, 5.37 [07:08:31] RECOVERY Current Load is now: OK on migration1 i-00000261 output: OK - load average: 0.27, 1.91, 3.30 [07:08:31] RECOVERY Current Users is now: OK on migration1 i-00000261 output: USERS OK - 0 users currently logged in [07:08:31] RECOVERY Disk Space is now: OK on migration1 i-00000261 output: DISK OK [07:08:31] RECOVERY Total Processes is now: OK on migration1 i-00000261 output: PROCS OK: 83 processes [07:08:36] PROBLEM Current Load is now: CRITICAL on gluster-devstack i-00000274 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:08:36] PROBLEM dpkg-check is now: CRITICAL on gluster-devstack i-00000274 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:09:05] PROBLEM Current Users is now: CRITICAL on gluster-devstack i-00000274 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:10:29] PROBLEM Current Load is now: CRITICAL on mobile-testing i-00000271 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:11:19] PROBLEM HTTP is now: WARNING on deployment-web3 i-00000219 output: HTTP WARNING: HTTP/1.1 403 Forbidden - 366 bytes in 0.014 second response time [07:13:06] PROBLEM Total Processes is now: CRITICAL on worker1 i-00000208 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:13:11] PROBLEM Disk Space is now: CRITICAL on worker1 i-00000208 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:13:11] PROBLEM Current Users is now: CRITICAL on worker1 i-00000208 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:13:11] PROBLEM Current Load is now: CRITICAL on worker1 i-00000208 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:13:11] PROBLEM Free ram is now: CRITICAL on upload-wizard i-0000021c output: CHECK_NRPE: Socket timeout after 10 seconds. [07:13:11] PROBLEM Free ram is now: CRITICAL on worker1 i-00000208 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:14:15] PROBLEM Current Load is now: WARNING on deployment-transcoding i-00000105 output: WARNING - load average: 6.56, 10.05, 6.83 [07:15:44] PROBLEM Disk Space is now: CRITICAL on incubator-bot1 i-00000251 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:16:05] PROBLEM Free ram is now: CRITICAL on incubator-bot1 i-00000251 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:16:09] "fun" [07:18:12] RECOVERY Disk Space is now: OK on worker1 i-00000208 output: DISK OK [07:18:12] RECOVERY Total Processes is now: OK on worker1 i-00000208 output: PROCS OK: 76 processes [07:18:17] RECOVERY Current Users is now: OK on worker1 i-00000208 output: USERS OK - 0 users currently logged in [07:18:17] PROBLEM Current Load is now: WARNING on worker1 i-00000208 output: WARNING - load average: 3.64, 6.09, 6.11 [07:18:17] RECOVERY Free ram is now: OK on worker1 i-00000208 output: OK: 94% free memory [07:18:22] PROBLEM Current Load is now: CRITICAL on incubator-bot2 i-00000252 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:20:00] PROBLEM HTTP is now: CRITICAL on deployment-web4 i-00000214 output: CRITICAL - Socket timeout after 10 seconds [07:20:00] PROBLEM HTTP is now: CRITICAL on deployment-web5 i-00000213 output: CRITICAL - Socket timeout after 10 seconds [07:20:19] PROBLEM HTTP is now: CRITICAL on deployment-web i-00000217 output: CRITICAL - Socket timeout after 10 seconds [07:20:37] PROBLEM Current Users is now: CRITICAL on precise-test i-00000231 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:20:37] PROBLEM Disk Space is now: CRITICAL on precise-test i-00000231 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:20:37] PROBLEM Total Processes is now: CRITICAL on precise-test i-00000231 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:20:42] PROBLEM dpkg-check is now: CRITICAL on precise-test i-00000231 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:20:42] PROBLEM Current Load is now: CRITICAL on precise-test i-00000231 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:20:42] PROBLEM Free ram is now: CRITICAL on precise-test i-00000231 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:20:48] PROBLEM Current Users is now: CRITICAL on rds i-00000207 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:20:48] PROBLEM Total Processes is now: CRITICAL on rds i-00000207 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:20:48] PROBLEM Free ram is now: CRITICAL on rds i-00000207 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:21:51] RECOVERY Current Users is now: OK on incubator-bot1 i-00000251 output: USERS OK - 0 users currently logged in [07:21:51] RECOVERY dpkg-check is now: OK on incubator-bot1 i-00000251 output: All packages OK [07:22:40] RECOVERY Current Load is now: OK on worker1 i-00000208 output: OK - load average: 0.08, 2.35, 4.48 [07:22:40] RECOVERY Current Load is now: OK on incubator-bot2 i-00000252 output: OK - load average: 0.42, 2.85, 4.02 [07:22:40] RECOVERY Free ram is now: OK on upload-wizard i-0000021c output: OK: 94% free memory [07:23:48] PROBLEM HTTP is now: WARNING on deployment-web4 i-00000214 output: HTTP WARNING: HTTP/1.1 403 Forbidden - 366 bytes in 0.017 second response time [07:23:48] PROBLEM HTTP is now: WARNING on deployment-web5 i-00000213 output: HTTP WARNING: HTTP/1.1 403 Forbidden - 366 bytes in 0.017 second response time [07:23:48] PROBLEM HTTP is now: WARNING on deployment-web i-00000217 output: HTTP WARNING: HTTP/1.1 403 Forbidden - 366 bytes in 0.017 second response time [07:25:25] RECOVERY Current Users is now: OK on precise-test i-00000231 output: USERS OK - 0 users currently logged in [07:28:47] RECOVERY Current Load is now: OK on deployment-transcoding i-00000105 output: OK - load average: 0.05, 1.33, 3.66 [07:29:35] PROBLEM Puppet freshness is now: CRITICAL on nova-ldap1 i-000000df output: Puppet has not run in last 20 hours [07:46:12] PROBLEM Current Load is now: WARNING on nagios 127.0.0.1 output: WARNING - load average: 0.71, 1.25, 3.93 [07:56:11] RECOVERY Current Load is now: OK on nagios 127.0.0.1 output: OK - load average: 0.87, 1.11, 2.53 [08:15:12] !log WMFLabs seems to have recovered now [08:15:13] WMFLabs is not a valid project. [08:25:12] o.o [08:25:21] morning hashar [08:25:24] :P [08:25:28] hello :) [08:26:21] hi petan|wk [08:26:26] which tz are you in anyway? [08:29:09] GMT [08:29:53] er, are you sure about that? :) [08:30:04] you probably aren't [08:33:46] paravoid: really? [08:33:55] where do you live? [08:33:58] europe [08:35:23] europe has many timezones [08:35:30] right this one has GMT+1 [08:35:33] that's like GMT [08:35:34] but very few countries are on GMT during summer time [08:35:46] ok right now I have 10:35 [08:35:47] right :-) [08:36:04] ok, so WEST [08:36:08] :-) [08:36:10] probably [08:36:32] I am in good tz to bother hashar whole day [08:36:59] 10:37 here [08:37:01] GMT + 2 [08:37:03] CET ? [08:37:09] oh, right, wo you're in CEST [08:37:11] not WEST [08:37:21] !hashar is [10:15:12] !log WMFLabs seems to have recovered now [08:37:25] Key was added! [08:37:26] you don't observe GMT even during winter [08:37:30] !ping [08:37:30] pong [08:37:52] WEST is probably portugal [08:38:04] yes [08:38:07] and WEST = BST [08:38:11] which is the UK too [09:11:46] PROBLEM Disk Space is now: WARNING on nova-production1 i-0000007b output: DISK WARNING - free space: / 573 MB (5% inode=80%): [09:29:02] !ping [09:29:02] pong [09:43:14] !log deployment-prep hashar: removed Draft extension for now. [09:43:16] Logged the message, Master [09:48:16] going to love that error [09:48:16] Exception thrown without a stack frame in Unknown on line 0 [09:48:18] \o/ [10:15:21] !log deployment-prep hashar: Fixed cluster which was giving blank page. Root cause was wmfUdp2logDest which must be (aka not use a hostname [10:15:22] Logged the message, Master [10:20:09] !log deployment-prep hashar: Fixed filerepo backend by using chown -R www-data:depops /mnt/export/upload6/wikibooks [10:20:10] Logged the message, Master [10:29:22] !log deployment-prep hashar: afterall, made /mnt/export/upload6 to be world writable "sudo chmod -R 777 *" [10:29:23] Logged the message, Master [10:48:51] RECOVERY Disk Space is now: OK on deployment-transcoding i-00000105 output: DISK OK [11:34:33] PROBLEM Puppet freshness is now: CRITICAL on hugglewiki i-000000aa output: Puppet has not run in last 20 hours [11:35:33] PROBLEM Puppet freshness is now: CRITICAL on wikistream-1 i-0000016e output: Puppet has not run in last 20 hours [11:46:42] PROBLEM Puppet freshness is now: CRITICAL on migration1 i-00000261 output: Puppet has not run in last 20 hours [12:00:36] PROBLEM Puppet freshness is now: CRITICAL on memcache-puppet i-00000153 output: Puppet has not run in last 20 hours [12:04:36] PROBLEM Puppet freshness is now: CRITICAL on jenkins2 i-00000102 output: Puppet has not run in last 20 hours [12:04:36] PROBLEM Puppet freshness is now: CRITICAL on publicdata-administration i-0000019e output: Puppet has not run in last 20 hours [12:04:36] PROBLEM Puppet freshness is now: CRITICAL on resourceloader2-apache i-000001d7 output: Puppet has not run in last 20 hours [12:05:36] PROBLEM Puppet freshness is now: CRITICAL on dumps-6 i-00000266 output: Puppet has not run in last 20 hours [12:05:36] PROBLEM Puppet freshness is now: CRITICAL on labs-nfs1 i-0000005d output: Puppet has not run in last 20 hours [12:05:36] PROBLEM Puppet freshness is now: CRITICAL on mailman-01 i-00000235 output: Puppet has not run in last 20 hours [12:05:36] PROBLEM Puppet freshness is now: CRITICAL on swift-be3 i-000001c9 output: Puppet has not run in last 20 hours [12:05:36] PROBLEM Puppet freshness is now: CRITICAL on wikidata-dev-3 i-00000225 output: Puppet has not run in last 20 hours [12:06:36] PROBLEM Puppet freshness is now: CRITICAL on wep i-000000c2 output: Puppet has not run in last 20 hours [12:07:41] PROBLEM Puppet freshness is now: CRITICAL on deployment-thumbproxy i-0000026b output: Puppet has not run in last 20 hours [12:07:41] PROBLEM Puppet freshness is now: CRITICAL on gerrit i-000000ff output: Puppet has not run in last 20 hours [12:07:41] PROBLEM Puppet freshness is now: CRITICAL on gluster-devstack i-00000274 output: Puppet has not run in last 20 hours [12:07:41]