[00:03:28] actually thinking about it - as long as I can identify the client from its cert I don't give a fuck, since if you have access to the client you can change the files the system would poll anyway [00:38:17] RECOVERY Free ram is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK: 22% free memory [00:38:57] RECOVERY Free ram is now: OK on swift-be4.pmtpa.wmflabs 10.4.0.127 output: OK: 20% free memory [00:39:33] Damianz: true [00:40:02] Damianz: that requires auth with the cert, though. whatever service you're writing to may not support it [00:40:13] unless you're writing an api in front of it :) [00:40:24] which honestly isn't insane [00:41:25] RECOVERY Free ram is now: OK on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: OK: 21% free memory [00:41:55] RECOVERY Free ram is now: OK on swift-be2.pmtpa.wmflabs 10.4.0.112 output: OK: 22% free memory [00:51:54] PROBLEM Free ram is now: WARNING on swift-be4.pmtpa.wmflabs 10.4.0.127 output: Warning: 18% free memory [00:59:58] PROBLEM Free ram is now: WARNING on swift-be2.pmtpa.wmflabs 10.4.0.112 output: Warning: 19% free memory [01:04:25] PROBLEM Free ram is now: WARNING on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: Warning: 12% free memory [01:06:25] PROBLEM Free ram is now: CRITICAL on sube.pmtpa.wmflabs 10.4.0.245 output: Connection refused by host [01:06:45] PROBLEM Total processes is now: WARNING on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: PROCS WARNING: 174 processes [01:07:15] PROBLEM Current Load is now: CRITICAL on sube.pmtpa.wmflabs 10.4.0.245 output: Connection refused by host [01:08:55] PROBLEM Total processes is now: CRITICAL on sube.pmtpa.wmflabs 10.4.0.245 output: Connection refused by host [01:09:55] RECOVERY Free ram is now: OK on swift-be2.pmtpa.wmflabs 10.4.0.112 output: OK: 23% free memory [01:16:45] RECOVERY Total processes is now: OK on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: PROCS OK: 96 processes [01:21:35] bah. I hardcoded sysadmin and netadmin [01:22:54] I wonder if keystone has calls for userCanDoX kind of things [01:22:58] or nova [01:27:37] PROBLEM Current Load is now: WARNING on ve-roundtrip2.pmtpa.wmflabs 10.4.0.162 output: WARNING - load average: 9.53, 7.68, 5.96 [01:29:17] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3.pmtpa.wmflabs 10.4.0.62 output: WARNING - load average: 7.73, 6.59, 5.47 [01:59:28] PROBLEM Total processes is now: WARNING on parsoid-roundtrip4-8core.pmtpa.wmflabs 10.4.0.39 output: PROCS WARNING: 152 processes [02:02:45] PROBLEM Current Load is now: WARNING on parsoid-roundtrip6-8core.pmtpa.wmflabs 10.4.0.222 output: WARNING - load average: 10.49, 8.89, 6.72 [02:02:55] PROBLEM Free ram is now: WARNING on swift-be2.pmtpa.wmflabs 10.4.0.112 output: Warning: 19% free memory [02:14:27] RECOVERY Total processes is now: OK on parsoid-roundtrip4-8core.pmtpa.wmflabs 10.4.0.39 output: PROCS OK: 148 processes [02:19:16] RECOVERY Current Load is now: OK on parsoid-roundtrip3.pmtpa.wmflabs 10.4.0.62 output: OK - load average: 2.41, 3.26, 4.89 [02:22:45] RECOVERY Current Load is now: OK on ve-roundtrip2.pmtpa.wmflabs 10.4.0.162 output: OK - load average: 2.06, 2.69, 4.70 [02:22:45] RECOVERY Current Load is now: OK on parsoid-roundtrip6-8core.pmtpa.wmflabs 10.4.0.222 output: OK - load average: 1.71, 2.50, 4.22 [02:41:36] RECOVERY Free ram is now: OK on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: OK: 25% free memory [02:45:46] PROBLEM Current Load is now: WARNING on ve-roundtrip2.pmtpa.wmflabs 10.4.0.162 output: WARNING - load average: 4.92, 5.38, 5.11 [02:59:41] PROBLEM Free ram is now: WARNING on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: Warning: 18% free memory [03:02:23] RECOVERY Current Load is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK - load average: 0.36, 0.10, 0.03 [03:03:53] RECOVERY Total processes is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: PROCS OK: 93 processes [03:06:27] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 10% free memory [03:30:34] PROBLEM Free ram is now: WARNING on bots-4.pmtpa.wmflabs 10.4.0.64 output: Warning: 17% free memory [03:37:55] RECOVERY Free ram is now: OK on swift-be2.pmtpa.wmflabs 10.4.0.112 output: OK: 22% free memory [04:04:43] RECOVERY Free ram is now: OK on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: OK: 20% free memory [04:05:55] PROBLEM Free ram is now: WARNING on swift-be2.pmtpa.wmflabs 10.4.0.112 output: Warning: 19% free memory [04:32:44] PROBLEM Free ram is now: WARNING on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: Warning: 19% free memory [04:39:25] RECOVERY Free ram is now: OK on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: OK: 21% free memory [04:40:57] RECOVERY Free ram is now: OK on swift-be2.pmtpa.wmflabs 10.4.0.112 output: OK: 22% free memory [04:41:27] RECOVERY Free ram is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK: 21% free memory [04:41:57] RECOVERY Free ram is now: OK on swift-be4.pmtpa.wmflabs 10.4.0.127 output: OK: 20% free memory [04:49:56] PROBLEM Free ram is now: WARNING on swift-be4.pmtpa.wmflabs 10.4.0.127 output: Warning: 17% free memory [04:52:23] PROBLEM Free ram is now: WARNING on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: Warning: 16% free memory [04:56:57] PROBLEM Total processes is now: CRITICAL on sube.pmtpa.wmflabs 10.4.0.245 output: Connection refused by host [04:59:27] PROBLEM Free ram is now: CRITICAL on sube.pmtpa.wmflabs 10.4.0.245 output: Connection refused by host [05:00:14] PROBLEM Current Load is now: CRITICAL on sube.pmtpa.wmflabs 10.4.0.245 output: Connection refused by host [05:09:46] RECOVERY Current Load is now: OK on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: OK - load average: 4.53, 4.78, 4.98 [05:18:55] PROBLEM Free ram is now: WARNING on swift-be2.pmtpa.wmflabs 10.4.0.112 output: Warning: 19% free memory [05:22:43] PROBLEM Current Load is now: WARNING on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: WARNING - load average: 5.14, 5.07, 5.03 [05:30:44] PROBLEM Free ram is now: WARNING on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: Warning: 19% free memory [05:37:45] RECOVERY Current Load is now: OK on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: OK - load average: 4.73, 4.87, 4.97 [05:54:24] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 16% free memory [05:55:17] RECOVERY Current Load is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK - load average: 0.07, 0.04, 0.01 [05:56:57] RECOVERY Total processes is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: PROCS OK: 93 processes [06:08:54] RECOVERY Free ram is now: OK on swift-be2.pmtpa.wmflabs 10.4.0.112 output: OK: 22% free memory [06:23:43] PROBLEM Free ram is now: WARNING on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: Warning: 19% free memory [06:26:58] PROBLEM Free ram is now: WARNING on swift-be2.pmtpa.wmflabs 10.4.0.112 output: Warning: 18% free memory [06:28:38] PROBLEM Total processes is now: WARNING on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: PROCS WARNING: 154 processes [06:31:13] PROBLEM dpkg-check is now: CRITICAL on fundraising-dev-awight.pmtpa.wmflabs 10.4.0.253 output: DPKG CRITICAL dpkg reports broken packages [06:32:23] PROBLEM Total processes is now: WARNING on parsoid-roundtrip4-8core.pmtpa.wmflabs 10.4.0.39 output: PROCS WARNING: 152 processes [06:36:15] RECOVERY dpkg-check is now: OK on fundraising-dev-awight.pmtpa.wmflabs 10.4.0.253 output: All packages OK [06:38:35] RECOVERY Total processes is now: OK on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: PROCS OK: 150 processes [06:39:25] PROBLEM Free ram is now: CRITICAL on sube.pmtpa.wmflabs 10.4.0.245 output: Connection refused by host [06:39:55] PROBLEM Total processes is now: CRITICAL on sube.pmtpa.wmflabs 10.4.0.245 output: Connection refused by host [06:43:16] PROBLEM Current Load is now: CRITICAL on sube.pmtpa.wmflabs 10.4.0.245 output: Connection refused by host [06:57:26] RECOVERY Total processes is now: OK on parsoid-roundtrip4-8core.pmtpa.wmflabs 10.4.0.39 output: PROCS OK: 148 processes [06:58:36] RECOVERY Free ram is now: OK on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: OK: 22% free memory [07:18:56] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 19% free memory [07:25:44] PROBLEM Current Load is now: WARNING on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: WARNING - load average: 5.57, 5.34, 5.12 [07:48:09] Where should requests to wiki sysops be placed? [07:48:53] weird https://labsconsole.wikimedia.org/wiki/Special:RecentChanges [07:50:44] RECOVERY Current Load is now: OK on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: OK - load average: 5.06, 4.93, 4.98 [07:57:15] PROBLEM dpkg-check is now: CRITICAL on dumps-1.pmtpa.wmflabs 10.4.0.218 output: DPKG CRITICAL dpkg reports broken packages [08:36:56] RECOVERY Free ram is now: OK on swift-be2.pmtpa.wmflabs 10.4.0.112 output: OK: 20% free memory [08:37:26] RECOVERY Free ram is now: OK on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: OK: 21% free memory [08:40:21] Nemo_bis which kind of requeust [08:43:28] petan: see RC or MediaWiki_talk:sidebar [08:43:40] I took a look on RC [08:43:45] but don't see anything :/ [08:43:56] ah lol [08:43:58] now I do [08:44:23] meh... [08:44:40] out of my competence lol ryan restricted editing of MediaWiki space to wmf people for some reason [08:44:53] I have sysop but I can't change it :/ [08:45:14] but... I can block Ryan! [08:45:17] :D [08:49:26] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 11% free memory [08:49:56] RECOVERY Total processes is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: PROCS OK: 93 processes [08:49:56] PROBLEM Free ram is now: WARNING on swift-be2.pmtpa.wmflabs 10.4.0.112 output: Warning: 18% free memory [08:53:23] RECOVERY Current Load is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK - load average: 0.00, 0.02, 0.00 [09:00:24] PROBLEM Free ram is now: WARNING on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: Warning: 16% free memory [09:19:25] PROBLEM Free ram is now: CRITICAL on sube.pmtpa.wmflabs 10.4.0.245 output: CHECK_NRPE: Error - Could not complete SSL handshake. [09:24:23] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 12% free memory [09:52:53] PROBLEM Total processes is now: CRITICAL on sube.pmtpa.wmflabs 10.4.0.245 output: Connection refused by host [09:56:15] PROBLEM Current Load is now: CRITICAL on sube.pmtpa.wmflabs 10.4.0.245 output: Connection refused by host [10:12:16] RECOVERY dpkg-check is now: OK on dumps-1.pmtpa.wmflabs 10.4.0.218 output: All packages OK [10:19:25] PROBLEM Free ram is now: CRITICAL on sube.pmtpa.wmflabs 10.4.0.245 output: Connection refused by host [11:09:56] RECOVERY Free ram is now: OK on swift-be2.pmtpa.wmflabs 10.4.0.112 output: OK: 22% free memory [11:11:31] hmm [11:21:15] RECOVERY Current Load is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK - load average: 0.22, 0.08, 0.02 [11:22:55] RECOVERY Total processes is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: PROCS OK: 93 processes [11:22:55] PROBLEM Free ram is now: WARNING on swift-be2.pmtpa.wmflabs 10.4.0.112 output: Warning: 18% free memory [11:24:25] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 10% free memory [12:37:54] RECOVERY Free ram is now: OK on swift-be2.pmtpa.wmflabs 10.4.0.112 output: OK: 21% free memory [12:39:24] RECOVERY Free ram is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK: 22% free memory [12:40:25] RECOVERY Free ram is now: OK on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: OK: 21% free memory [12:53:26] PROBLEM Free ram is now: WARNING on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: Warning: 16% free memory [12:55:55] PROBLEM Free ram is now: WARNING on swift-be2.pmtpa.wmflabs 10.4.0.112 output: Warning: 18% free memory [12:57:25] PROBLEM Free ram is now: CRITICAL on sube.pmtpa.wmflabs 10.4.0.245 output: Connection refused by host [12:59:22] PROBLEM Current Load is now: CRITICAL on sube.pmtpa.wmflabs 10.4.0.245 output: Connection refused by host [13:00:54] PROBLEM Total processes is now: CRITICAL on sube.pmtpa.wmflabs 10.4.0.245 output: Connection refused by host [13:30:44] PROBLEM Free ram is now: CRITICAL on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: Critical: 5% free memory [13:40:55] RECOVERY Free ram is now: OK on swift-be2.pmtpa.wmflabs 10.4.0.112 output: OK: 22% free memory [13:54:14] RECOVERY Current Load is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK - load average: 0.28, 0.09, 0.03 [13:55:57] RECOVERY Total processes is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: PROCS OK: 93 processes [13:57:27] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 10% free memory [14:13:55] PROBLEM Free ram is now: WARNING on swift-be2.pmtpa.wmflabs 10.4.0.112 output: Warning: 18% free memory [15:49:25] PROBLEM Disk Space is now: WARNING on syslogcol-ab.pmtpa.wmflabs 10.4.0.223 output: DISK WARNING - free space: / 573 MB (5% inode=90%): [16:08:58] RECOVERY Free ram is now: OK on swift-be2.pmtpa.wmflabs 10.4.0.112 output: OK: 21% free memory [16:26:35] PROBLEM Free ram is now: WARNING on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: Warning: 19% free memory [16:26:55] PROBLEM Free ram is now: WARNING on swift-be2.pmtpa.wmflabs 10.4.0.112 output: Warning: 17% free memory [16:31:44] RECOVERY Free ram is now: OK on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: OK: 20% free memory [16:37:24] RECOVERY Free ram is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK: 22% free memory [16:38:24] RECOVERY Free ram is now: OK on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: OK: 21% free memory [16:41:57] RECOVERY Free ram is now: OK on swift-be2.pmtpa.wmflabs 10.4.0.112 output: OK: 21% free memory [16:51:24] PROBLEM Free ram is now: WARNING on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: Warning: 15% free memory [17:02:23] PROBLEM Current Load is now: CRITICAL on sube.pmtpa.wmflabs 10.4.0.245 output: Connection refused by host [17:03:53] PROBLEM Total processes is now: CRITICAL on sube.pmtpa.wmflabs 10.4.0.245 output: Connection refused by host [17:04:53] RECOVERY Free ram is now: OK on swift-be4.pmtpa.wmflabs 10.4.0.127 output: OK: 21% free memory [17:05:27] PROBLEM Free ram is now: CRITICAL on sube.pmtpa.wmflabs 10.4.0.245 output: Connection refused by host [17:08:57] PROBLEM Current Load is now: CRITICAL on mwreview-testing.pmtpa.wmflabs 10.4.0.82 output: Connection refused by host [17:09:37] PROBLEM Disk Space is now: CRITICAL on mwreview-testing.pmtpa.wmflabs 10.4.0.82 output: Connection refused by host [17:10:25] PROBLEM Free ram is now: CRITICAL on mwreview-testing.pmtpa.wmflabs 10.4.0.82 output: Connection refused by host [17:11:45] PROBLEM Total processes is now: CRITICAL on mwreview-testing.pmtpa.wmflabs 10.4.0.82 output: Connection refused by host [17:12:25] PROBLEM dpkg-check is now: CRITICAL on mwreview-testing.pmtpa.wmflabs 10.4.0.82 output: Connection refused by host [17:12:55] PROBLEM Free ram is now: WARNING on swift-be4.pmtpa.wmflabs 10.4.0.127 output: Warning: 16% free memory [17:13:56] RECOVERY Current Load is now: OK on mwreview-testing.pmtpa.wmflabs 10.4.0.82 output: OK - load average: 1.00, 1.00, 0.54 [17:14:35] RECOVERY Disk Space is now: OK on mwreview-testing.pmtpa.wmflabs 10.4.0.82 output: DISK OK [17:15:25] RECOVERY Free ram is now: OK on mwreview-testing.pmtpa.wmflabs 10.4.0.82 output: OK: 541% free memory [17:16:45] RECOVERY Total processes is now: OK on mwreview-testing.pmtpa.wmflabs 10.4.0.82 output: PROCS OK: 91 processes [17:17:25] RECOVERY dpkg-check is now: OK on mwreview-testing.pmtpa.wmflabs 10.4.0.82 output: All packages OK [17:35:02] PROBLEM Free ram is now: WARNING on swift-be2.pmtpa.wmflabs 10.4.0.112 output: Warning: 18% free memory [18:15:35] PROBLEM Free ram is now: WARNING on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: Warning: 17% free memory [18:24:15] PROBLEM Free ram is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: Warning: 6% free memory [18:29:06] Is Ryan Lane around? [18:29:18] PROBLEM Free ram is now: CRITICAL on bots-cb.pmtpa.wmflabs 10.4.0.44 output: Critical: 2% free memory [18:32:09] <^demon> Krenair: Maybe now? :) [18:32:15] I saw :) [18:34:13] RECOVERY Free ram is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: OK: 322% free memory [18:39:54] RECOVERY Free ram is now: OK on swift-be2.pmtpa.wmflabs 10.4.0.112 output: OK: 21% free memory [18:48:13] andrewbogott, hey is that your code sitting in nova-precise2:/srv/org/wikimedia/controller/wikis/w/extensions/OpenStackManager ? [18:48:44] Krenair: Some of it, at least. I'm working on SpecialNovaSudoer.php [18:49:07] I'm adding additional wikis now [18:49:26] I just tried to checkout master and noticed a change. I went back so it should be as you left it, sorry :) [18:49:47] No worries -- I'm editing elsewhere and then just dropping in the files for test runs. [18:58:59] RECOVERY Total processes is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: PROCS OK: 93 processes [19:00:25] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 10% free memory [19:02:23] RECOVERY Current Load is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK - load average: 0.00, 0.04, 0.01 [19:02:53] PROBLEM Free ram is now: WARNING on swift-be2.pmtpa.wmflabs 10.4.0.112 output: Warning: 17% free memory [19:22:36] Ryan_Lane: Hey, I'm having some trouble accessing orgcharts-dev.pmtpa.wmflabs, is there anything relevant happening? [19:22:48] no [19:23:19] I can ssh into it [19:23:36] is apache supposed to be running? [19:23:53] ugh [19:23:56] god damn it [19:24:00] this is oneiric? :( [19:24:04] Hm, probably not, it doesn't have anything running through apache [19:24:14] Ryan_Lane: I haven't touched it in a while, so probably yes [19:24:24] PROBLEM dpkg-check is now: CRITICAL on lucene-test1.pmtpa.wmflabs 10.4.0.205 output: DPKG CRITICAL dpkg reports broken packages [19:24:40] oneiric instances are mostly broken [19:24:57] Ryan_Lane: Well, upgrading can't make it any worse [19:24:59] let me see if I can upgrade gluster using a deprecated ppa [19:25:00] no [19:25:00] no [19:25:04] never, ever upgrade [19:25:23] I even went as far as removing the upgrade tool [19:25:30] you'll permanently break the instance [19:25:38] Ryan_Lane: Right, well, I'm listening [19:25:52] I'll try to get this working [19:25:56] *nod* thanks [19:25:57] but you need to create a new instance [19:26:01] Ah. [19:26:02] and move your stuff to it [19:26:06] which is why puppet is recommended [19:26:08] This is doable [19:28:47] hm. weird. i replaced the gluster package and the instance stopped responding. [19:28:56] let me reboot it [19:29:27] Fun times [19:29:38] Ryan_Lane: If you can't get it up again it would be OK, I have backups of everything [19:29:59] well, I think a gluster process probably oops the kernel [19:30:23] I probably should have killed those before replacing its package [19:30:48] oh [19:30:49] wait [19:30:53] it's in a rebooting state? [19:30:59] did you reboot it? [19:31:20] I tried to, but it didn't appear to change anything [19:31:34] I mean just a little bit ago [19:31:39] like a few minutes ago [19:31:56] Yes [19:32:00] ah [19:32:02] Before I pinged you [19:32:06] that's why it rebooted on me [19:32:07] heh [19:32:10] Indeed [19:32:13] it probably just took that long to go through [19:33:54] PROBLEM Current Load is now: CRITICAL on orgchart.pmtpa.wmflabs 10.4.0.86 output: Connection refused by host [19:33:54] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [19:34:23] Ooh, fun times. [19:34:25] RECOVERY dpkg-check is now: OK on lucene-test1.pmtpa.wmflabs 10.4.0.205 output: All packages OK [19:34:34] PROBLEM Disk Space is now: CRITICAL on orgchart.pmtpa.wmflabs 10.4.0.86 output: Connection refused by host [19:35:15] PROBLEM Free ram is now: CRITICAL on orgchart.pmtpa.wmflabs 10.4.0.86 output: Connection refused by host [19:35:59] Hrm, new instance seems to be unhappy with me [19:36:40] it takes a bit to finish building [19:36:45] Righto. [19:36:45] PROBLEM Total processes is now: CRITICAL on orgchart.pmtpa.wmflabs 10.4.0.86 output: Connection refused by host [19:36:46] puppet has to run [19:36:55] and a full puppet run takes 6 entire minutes, which is absurd [19:37:09] I guess I can just wait for the "RECOVERY" message [19:37:29] we're working on adding an echo notification when it finishes [19:37:45] Krenair is, specifically :) [19:37:45] https://gerrit.wikimedia.org/r/#/c/45277 [19:37:52] Good plan [19:38:55] RECOVERY Current Load is now: OK on orgchart.pmtpa.wmflabs 10.4.0.86 output: OK - load average: 0.27, 0.78, 0.51 [19:39:01] Sweet. [19:39:35] RECOVERY Disk Space is now: OK on orgchart.pmtpa.wmflabs 10.4.0.86 output: DISK OK [19:40:15] RECOVERY Free ram is now: OK on orgchart.pmtpa.wmflabs 10.4.0.86 output: OK: 739% free memory [19:41:45] RECOVERY Total processes is now: OK on orgchart.pmtpa.wmflabs 10.4.0.86 output: PROCS OK: 92 processes [19:46:18] Ryan_Lane: It's possible I won't need to recover the other instance, most of the data was in /data/project anyway [19:46:24] ah [19:46:24] ok [19:46:42] well, I stupidly rebooted it again while it was coming up [19:47:05] I'm pretty sure it'll come back up [19:47:20] if you get it going again before I'm done, let me know and we can just delete this instance [19:48:31] Righto [19:48:50] I'm in the process of bringing it up, shouldn't take too long [19:50:03] Argh, the locale settings again [19:51:16] heh [19:51:20] (puppet) [19:52:53] Ryan_Lane: Is there some magic to setting the locale that I'm not getting? [19:53:12] what's wrong with the locale? [19:53:14] Aha, a wiki page about it [19:53:22] https://labsconsole.wikimedia.org/wiki/Help:Locale_errors [20:04:24] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [20:06:25] Ryan_Lane, Author: Translation updater bot ? [20:09:59] Ryan_Lane: FYI I haven't been able to get the data loaded yet, but it's probably fine if I load the backup that I have from December. I could use access to the old instance to figure out the configuration I used to make this work, but if you can't get it back up then I can make do [20:34:24] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [20:36:42] :q [20:37:55] RECOVERY Free ram is now: OK on swift-be2.pmtpa.wmflabs 10.4.0.112 output: OK: 20% free memory [20:38:13] Krenair: where do you see that? [20:38:42] https://gerrit.wikimedia.org/r/#/c/45290/ [20:39:42] hahaha [20:39:49] I fucked up an amended commit :) [20:39:54] let me fix that :) [20:40:27] PROBLEM Free ram is now: CRITICAL on sube.pmtpa.wmflabs 10.4.0.245 output: Connection refused by host [20:41:25] Krenair: wait. that still isn't riht [20:41:26] right [20:41:27] RECOVERY Free ram is now: OK on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: OK: 21% free memory [20:41:32] unicorn-poop deployer.. sounds good [20:41:52] Well at least you're not a bot now :) [20:41:53] Krenair: can you amend the patchset? use: git commit --amend —reset-author [20:42:01] you're the author ;) [20:42:25] Huh? We're talking about https://gerrit.wikimedia.org/r/#/c/45290/ [20:42:41] oh [20:42:42] wai [20:42:44] *wait [20:42:44] right [20:42:48] my mind isn't working today [20:42:51] :D [20:43:15] thanks for catching that :D [20:45:14] PROBLEM Current Load is now: CRITICAL on sube.pmtpa.wmflabs 10.4.0.245 output: Connection refused by host [20:46:42] * Damianz feeds ryan's brain some cocaine [20:46:54] PROBLEM Total processes is now: CRITICAL on sube.pmtpa.wmflabs 10.4.0.245 output: Connection refused by host [20:47:04] marktraceur: I can't get the instance back up [20:47:09] marktraceur: I can pull data from it, though [20:47:14] marktraceur: what do you need? [20:49:42] Ryan_Lane: The mongodb files that are in (probably) /var/lib/mongodb [20:49:53] ok. one sec [20:50:11] The crontab would also be useful [20:50:11] I see some files there [20:50:38] Ryan_Lane: If you could put them all in /data/project/mongodb I could use that from now on [20:50:49] on the new instance? [20:50:57] Well, in the project storage [20:51:02] ah. right [20:51:02] ok [20:51:53] marktraceur: anything else? [20:52:09] Uhm, that should do it, I think [20:52:12] ok [20:52:15] Let me verify things are working [20:52:28] Oh, not done yet? [20:52:47] need something else? [20:53:15] I just need to re-tar, if so [20:53:16] No, I don't see the mongodb directory in /data/project [20:53:20] Oh, OK [20:53:29] I need to pull this off the host [20:53:35] then scp it to your instance [20:53:46] Yes, I see now, I'm good with crontab and the mongodb files [20:53:59] what's the new instance name? [20:54:21] ah. orgchart [20:54:25] Yup. [20:54:54] done [20:56:35] Sweet. [20:56:48] <^demon> Ryan_Lane: Oh btw, https://gerrit.wikimedia.org/r/#/c/45332/ <-- Less svn crap \o/ [20:57:50] merged [21:04:24] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [21:04:25] PROBLEM Free ram is now: WARNING on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: Warning: 16% free memory [21:04:47] Ryan_Lane, I just created a db on bots-sql3 but apparently after creating it I wast given any privileges for it, is there any chance you could pop in and allow me to use it? :) [21:05:01] how did you create it? [21:05:23] phpmyadmin ;p, I'm guessing that is where I went wrong? [21:05:26] I'd imagine if you have create rights, you probably have credentials [21:05:35] phpmyadmin? where's that running? [21:05:40] I thought we killed that off [21:05:50] well, http://bots.wmflabs.org/phpmyadmin [21:05:51] works [21:05:54] phpmyadmin is always so security bug ridden [21:06:05] <^demon> I suggested pinning it to a negative value or somesuch idea. [21:06:06] It allowed me to create a db and a table but not to access anything :P [21:06:33] Ryan_Lane: Thanks so much, things are working again [21:06:34] great [21:06:34] yw [21:06:55] ^demon: yeah. [21:07:04] <^demon> Package: phpmyadmin [21:07:10] <^demon> Pin-Priority: -100 [21:07:43] <^demon> Or -1, anything < 0 works supposedly. [21:07:46] addshore: I'm not sure how the database stuff is supposed to be working or used [21:08:00] addshore: it's likely best to talk with Damianz and/or petan [21:08:47] okey dokey :) [21:10:14] urgh [21:10:18] RECOVERY Current Load is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK - load average: 0.23, 0.06, 0.02 [21:10:18] I'm gonna fucking stab whoever installed pma [21:10:28] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 16% free memory [21:10:28] xD [21:10:37] not me :> [21:11:09] It's exploited 3 servers so far in bots... which I had to clean up, there's a reason I spent a while writing puppet stuff, testing configs and planning upgrades then other people just do it [21:11:13] * Damianz rant [21:11:15] grr [21:12:01] RECOVERY Total processes is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: PROCS OK: 93 processes [21:12:01] Ryan_Lane: Ask basically since sql are sudo restricted... most accounts are far too open - we need to lock them down and form a standard but meh [21:12:18] PROBLEM Free ram is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: Warning: 8% free memory [21:12:34] that's on apache01, right? [21:12:47] that doesn't allow root, does it? [21:13:14] Damianz, how can I go about droping or gaining access to my db? :) [21:13:22] yes [21:13:31] I'm gonna re-install it back to my puppet class and remove access [21:13:32] we allow root on apache01? [21:13:37] yes [21:13:39] ugh [21:13:40] why? [21:13:44] no idea [21:13:49] someone rebuilt /my/ apache box [21:13:49] let's remove that [21:13:53] which was all nice [21:13:55] and secure [21:13:57] and automated [21:14:00] -.- [21:14:04] -_- [21:14:09] >.< [21:14:32] fuck it [21:14:38] I'll wait until puppet gets done and merged [21:14:56] might as well do it once [21:15:09] Ryan_Lane: ? [21:15:14] yes? [21:15:43] can you open a port (e.g. 9001) on my instance, so that I can also test my Etherpad Lite there [21:15:54] why are you running etherpad there? [21:16:02] it's the openid project ;) [21:16:10] Because I can [21:16:35] If you like [21:16:36] what are you going to use it for? [21:16:50] testing it and my extension there, so that [21:16:55] everyone can see how it works [21:17:07] that's what the etherpad project is for [21:17:16] RECOVERY Free ram is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: OK: 282% free memory [21:17:16] yes, i saw that [21:17:30] also, don't use port 9001 [21:17:30] do you have defined me correctly there, too ? [21:17:38] EPL runs on node [21:17:42] and node needs a prot [21:17:44] port [21:17:51] I am sure, you know this [21:18:00] yes, but you can use a reverse proxy [21:18:00] reverse proxy it [21:18:01] if you don'rt like me to test it, I will leave [21:18:03] so that it's on 80 [21:18:11] reverse proxy can be evil [21:18:15] Can I [21:18:16] what now/ [21:18:17] ? [21:18:22] how is it evil? [21:18:36] reverse proxy can be evil [21:18:42] *how*? [21:18:43] evil is installing php my admin on a lovely server [21:18:46] if the server has holes [21:18:50] ummm [21:19:04] I really don't understand what you mean [21:19:06] phpmyadmin without https:// and authentication [21:19:24] you'd be reverse proxying something specific to something specific [21:19:53] Wikinaut: you are already a member of the etherpad project and you're a sysadmin in it [21:19:54] http://seclists.org/fulldisclosure/2011/Oct/232 [21:20:01] use that for etherpad, not openid [21:20:04] ok [21:20:10] Damianz, any chance you could fix my mess? :) [21:20:11] and use a reverse proxy for port 9001 [21:20:16] yep [21:20:19] I do know, how [21:20:26]