[00:01:54] RECOVERY Current Load is now: OK on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: OK - load average: 3.33, 3.88, 4.71 [00:02:26] Sometimes I'll make a change on a labs instance, copy it to a 3 1/2-inch floppy and bring it back to my MacBook, then, ??? [00:05:52] RECOVERY Current Load is now: OK on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: OK - load average: 6.09, 4.47, 4.91 [00:09:32] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [00:18:52] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: WARNING - load average: 10.24, 7.82, 6.05 [00:19:52] PROBLEM Current Load is now: WARNING on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: WARNING - load average: 6.71, 6.33, 5.40 [00:24:52] RECOVERY Current Load is now: OK on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: OK - load average: 4.09, 4.77, 5.00 [00:39:32] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [00:42:54] PROBLEM Free ram is now: WARNING on dumps-bot2 i-000003f4.pmtpa.wmflabs output: Warning: 19% free memory [00:56:52] PROBLEM Current Load is now: WARNING on parsoid-roundtrip5-8core i-000004db.pmtpa.wmflabs output: WARNING - load average: 7.19, 6.37, 5.52 [01:02:53] PROBLEM Current Load is now: WARNING on parsoid-roundtrip4 i-000004d7.pmtpa.wmflabs output: WARNING - load average: 5.74, 6.65, 5.48 [01:06:52] PROBLEM Total processes is now: WARNING on bots-salebot i-00000457.pmtpa.wmflabs output: PROCS WARNING: 177 processes [01:09:32] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [01:11:53] RECOVERY Total processes is now: OK on bots-salebot i-00000457.pmtpa.wmflabs output: PROCS OK: 101 processes [01:28:38] PROBLEM Disk Space is now: WARNING on mw1-21beta-lucid i-00000416.pmtpa.wmflabs output: DISK WARNING - free space: / 78 MB (5% inode=51%): [01:33:32] RECOVERY Current Users is now: OK on parsoid-spof i-000004d6.pmtpa.wmflabs output: USERS OK - 0 users currently logged in [01:38:44] RECOVERY Disk Space is now: OK on mw1-21beta-lucid i-00000416.pmtpa.wmflabs output: DISK OK [01:39:32] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [01:41:34] PROBLEM Current Users is now: WARNING on parsoid-spof i-000004d6.pmtpa.wmflabs output: USERS WARNING - 6 users currently logged in [01:56:42] PROBLEM Disk Space is now: WARNING on mw1-21beta-lucid i-00000416.pmtpa.wmflabs output: DISK WARNING - free space: / 78 MB (5% inode=51%): [01:58:22] PROBLEM Total processes is now: WARNING on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS WARNING: 153 processes [02:05:52] PROBLEM Current Load is now: WARNING on parsoid-spof i-000004d6.pmtpa.wmflabs output: WARNING - load average: 1.12, 8.19, 8.64 [02:07:52] RECOVERY Current Load is now: OK on parsoid-roundtrip4 i-000004d7.pmtpa.wmflabs output: OK - load average: 4.29, 4.33, 4.84 [02:09:32] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [02:15:53] RECOVERY Current Load is now: OK on parsoid-spof i-000004d6.pmtpa.wmflabs output: OK - load average: 0.95, 1.89, 4.97 [02:23:52] RECOVERY Current Load is now: OK on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: OK - load average: 2.03, 2.72, 4.62 [02:26:52] RECOVERY Current Load is now: OK on parsoid-roundtrip5-8core i-000004db.pmtpa.wmflabs output: OK - load average: 1.01, 1.72, 4.03 [02:38:42] RECOVERY Free ram is now: OK on bots-sql2 i-000000af.pmtpa.wmflabs output: OK: 20% free memory [02:39:32] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [02:40:42] RECOVERY Free ram is now: OK on bots-3 i-000000e5.pmtpa.wmflabs output: OK: 21% free memory [02:51:42] PROBLEM Free ram is now: WARNING on bots-sql2 i-000000af.pmtpa.wmflabs output: Warning: 14% free memory [02:53:44] PROBLEM Free ram is now: WARNING on bots-3 i-000000e5.pmtpa.wmflabs output: Warning: 18% free memory [03:05:52] PROBLEM Current Load is now: WARNING on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: WARNING - load average: 6.08, 8.05, 5.41 [03:09:36] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [03:09:52] PROBLEM Current Load is now: WARNING on parsoid-roundtrip5-8core i-000004db.pmtpa.wmflabs output: WARNING - load average: 10.67, 9.94, 7.07 [03:11:52] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: WARNING - load average: 7.04, 7.52, 5.93 [03:15:55] RECOVERY Current Load is now: OK on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: OK - load average: 0.25, 2.23, 3.80 [03:19:53] RECOVERY Current Load is now: OK on parsoid-roundtrip5-8core i-000004db.pmtpa.wmflabs output: OK - load average: 2.28, 3.34, 4.88 [03:36:53] RECOVERY Current Load is now: OK on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: OK - load average: 2.09, 2.66, 4.34 [03:39:37] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [03:58:42] RECOVERY Free ram is now: OK on bots-3 i-000000e5.pmtpa.wmflabs output: OK: 20% free memory [04:01:52] PROBLEM Free ram is now: WARNING on orgcharts-dev i-0000018f.pmtpa.wmflabs output: Warning: 15% free memory [04:09:33] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [04:21:43] PROBLEM Free ram is now: WARNING on bots-3 i-000000e5.pmtpa.wmflabs output: Warning: 17% free memory [04:21:53] PROBLEM Free ram is now: CRITICAL on orgcharts-dev i-0000018f.pmtpa.wmflabs output: Critical: 5% free memory [04:23:33] RECOVERY Total processes is now: OK on aggregator-test1 i-000002bf.pmtpa.wmflabs output: PROCS OK: 149 processes [04:31:32] PROBLEM Total processes is now: WARNING on aggregator-test1 i-000002bf.pmtpa.wmflabs output: PROCS WARNING: 153 processes [04:31:54] RECOVERY Free ram is now: OK on orgcharts-dev i-0000018f.pmtpa.wmflabs output: OK: 95% free memory [04:39:33] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [05:09:42] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [05:39:43] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [05:57:49] Hi guys [05:57:53] You guys r0ck [06:11:25] 10/23/2012 - 06:11:25 - User louisdang may have been modified in LDAP or locally, updating key in project(s): hadoop [06:11:42] 10/23/2012 - 06:11:42 - Updating keys for louisdang at /export/keys/louisdang [06:12:02] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [06:42:05] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [07:12:03] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [07:22:46] good morning [07:26:32] RECOVERY Current Users is now: OK on parsoid-spof i-000004d6.pmtpa.wmflabs output: USERS OK - 1 users currently logged in [07:42:52] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [08:13:52] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [08:19:17] !log wikidata-dev wikidata-dev4: installed puppetmaster::self to prepare a Wikidata installation from puppet [08:19:18] Logged the message, Master [08:43:52] PROBLEM Current Load is now: WARNING on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: WARNING - load average: 7.15, 6.38, 5.43 [08:44:12] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [09:14:14] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [09:24:40] Ryan_Lane hi [09:34:53] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: WARNING - load average: 6.45, 6.67, 5.53 [09:42:52] PROBLEM Current Load is now: WARNING on parsoid-roundtrip5-8core i-000004db.pmtpa.wmflabs output: WARNING - load average: 3.97, 5.04, 5.03 [09:44:15] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [09:56:52] PROBLEM Current Load is now: WARNING on parsoid-roundtrip4 i-000004d7.pmtpa.wmflabs output: WARNING - load average: 6.19, 5.94, 5.36 [10:01:01] !log deployment-prep j: update DocumentRoot of upload.beta.wmflabs.org to /mnt/upload7 in /usr/local/apache/conf/upload.conf (fixes 403 for videos) [10:01:01] Logged the message, Master [10:14:13] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [10:16:52] RECOVERY Current Load is now: OK on parsoid-roundtrip4 i-000004d7.pmtpa.wmflabs output: OK - load average: 4.35, 4.62, 4.95 [10:29:53] PROBLEM Free ram is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Critical: 5% free memory [10:39:55] RECOVERY Current Load is now: OK on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: OK - load average: 4.41, 4.30, 4.85 [10:44:13] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [10:52:55] RECOVERY Current Load is now: OK on parsoid-roundtrip5-8core i-000004db.pmtpa.wmflabs output: OK - load average: 3.24, 3.58, 4.80 [10:53:23] RECOVERY Total processes is now: OK on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS OK: 150 processes [10:53:53] RECOVERY Current Load is now: OK on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: OK - load average: 3.31, 3.69, 4.93 [11:14:22] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [11:44:22] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [12:14:24] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [12:16:43] PROBLEM Free ram is now: CRITICAL on bots-3 i-000000e5.pmtpa.wmflabs output: Critical: 5% free memory [12:26:42] RECOVERY Free ram is now: OK on bots-3 i-000000e5.pmtpa.wmflabs output: OK: 73% free memory [12:31:33] RECOVERY Total processes is now: OK on aggregator-test1 i-000002bf.pmtpa.wmflabs output: PROCS OK: 150 processes [12:44:23] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [13:05:34] PROBLEM Total processes is now: WARNING on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS WARNING: 196 processes [13:10:35] RECOVERY Total processes is now: OK on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS OK: 103 processes [13:14:27] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [13:44:22] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [13:56:47] !htmllogs [13:56:47] experimental: http://bots.wmflabs.org/~wm-bot/html/%23wikimedia-labs [14:14:23] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [14:44:35] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [15:04:32] PROBLEM Total processes is now: WARNING on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS WARNING: 196 processes [15:14:32] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [15:23:22] PROBLEM Total processes is now: WARNING on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS WARNING: 155 processes [15:24:32] PROBLEM Current Users is now: WARNING on parsoid-spof i-000004d6.pmtpa.wmflabs output: USERS WARNING - 6 users currently logged in [15:29:34] RECOVERY Total processes is now: OK on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS OK: 103 processes [15:39:46] andrewbogott: I reviewed your chang, do you want to fix it or should I? [15:40:38] Jan_Luca: I can do it. But, regarding the #includes... [15:40:52] Well, basically I don't understand when #includes are needed. Do you? [15:40:52] PROBLEM Current Load is now: WARNING on parsoid-roundtrip5-8core i-000004db.pmtpa.wmflabs output: WARNING - load average: 7.34, 6.83, 5.58 [15:41:05] The class works as is, but maybe that's somehow coincidental. [15:42:33] Hm… I should retest. It looked like it was working but now I feel like it shouldn't have been [15:43:00] Talking about puppet ... :D [15:43:28] Maybe there is a include "*.pp" somewhere ... [15:44:30] If I use the mediawiki-labs puppet stuff and then want to put Wikibase extensions into that installation -> How am I supposed to modify LocalSettings.php? [15:44:31] Yeah, I think everything is included. But maybe it's still more polite to explicitly include things... [15:44:32] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [15:44:53] All my changes are thrown into the file bucket. :´( [15:45:10] Quote: "Puppet has an import keyword for importing other manifests. You should almost never use it, as almost every use case for it has been replaced by the -----> module autoloader <-----." [15:45:35] Jan_Luca: That's definitive! [15:45:36] This seems to be the reason [15:45:55] Silke_WMDE: To quote that very file: "If you need to hand-edit local settings, modify the included orig/LocalSettings.php" [15:46:05] Silke_WMDE: You have to use /srv/mediawiki/orig/LocalSettings.php [15:48:13] Hm. But I would then controll that file via puppet. Is that how I am supposed to do it? [15:48:39] (Because then, it's no longer a real *local* changes file.) [15:49:12] Then you cannot use the standard puppet-class [15:50:06] Silke_WMDE: You want to add additional extensions via puppet? Or 'by hand' on the instance? [15:50:06] via puppet [15:50:06] Ah, OK. [15:50:27] I would like puppet to produce ready made wikidata servers. [15:50:37] In that case probably you'll want to create a new role class. And you can either modify the existing LocalSettings.php template or create a new one to go with your new class. [15:50:57] Creating a new role is about to become much easier as soon as Jan and I agree on this in-progress refactor. [15:51:18] Ah ok. [15:51:35] Silke_WMDE: Did what I said about templates make sense? [15:51:38] Is it possible to include a class and overwrite the "file { '/srv/mediawiki/LocalSettings.php': ..."-statement [15:52:35] Jan_Luca: I don't think so -- but that path could be parameterized. [15:52:59] Jan_Luca: No, thats what I tried [15:53:19] andrewbogott: templates? [15:54:19] Silke_WMDE: Just, do you understand roughtly where the puppetized LocalSettings.php is coming from? It's a template file, it lives in templates/mediawiki [15:54:34] yes [15:54:39] andrewbogott: It is possible: https://groups.google.com/forum/?fromgroups=#!topic/puppet-users/lEtx6xiSQD0 [15:54:39] And contains some logic to modify it at generation time. [15:55:01] yes I understand (at leasts more or less) [15:56:47] Jan_Luca: Hm… that seems worse than adding a param [15:57:09] I don't say it is better than a parameter ;-) [15:57:09] Jan_Luca: Crap, so the git::clone 'latest' behavior doesn't work unless the repo is already checked out? [15:58:14] andrewbogott: It will be first checked out and then it uses git pull [15:59:30] I'll have a go with a new role class. (You might hear from me again...) Thanks for the help! [15:59:55] It should fetch/rebase instead of pulling, but that's a question for mark [16:00:52] RECOVERY Current Load is now: OK on parsoid-roundtrip5-8core i-000004db.pmtpa.wmflabs output: OK - load average: 4.37, 4.49, 4.94 [16:14:33] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [16:19:22] andrewbogott: git pull is another cmd for "git fetch && git rebase FETCH_HEAD" [16:20:33] I don't think so. rebase puts upstream changes before local changes, I believe pull does the opposite. [16:20:36] http://stackoverflow.com/questions/3357122/git-pull-vs-git-fetch-git-rebase [16:21:30] Sorry: Quote: "In its default mode, git pull is shorthand for git fetch followed by git merge FETCH_HEAD" [16:21:31] Downstream reps should rebase and upstream reps should pull. But since our upstream reps are managed by gerrit, I'm pretty sure no one should be using pull ever. [16:21:50] For the rebase you have to use "git pull --rebase" [16:22:09] * andrewbogott nods [16:22:43] If you only want to get the changes made in Gerrit and do not change the code locally, you can use git pull [16:23:01] Sure. [16:23:20] Oh, wait, that page says… 'Everyone who thinks git pull --rebase === git fetch && git rebase is missing a big part of the story that will come back to bite them some day.' [16:24:30] When you have local changes I use a seperate branch, then git pull in the master branch and in the branch with changes then git rebase master [16:26:08] Jan_Luca: That's clearly correct :) But if you follow my rule of 'never use pull' then you don't have to pay attention to branch distinctions. [16:37:48] Change on 12mediawiki a page Developer access was modified, changed by Prageck link https://www.mediawiki.org/w/index.php?diff=596729 edit summary: [16:41:22] 10/23/2012 - 16:41:20 - Creating a home directory for jimmy at /export/keys/jimmy [16:44:33] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [16:46:18] 10/23/2012 - 16:46:18 - Updating keys for jimmy at /export/keys/jimmy [16:53:52] PROBLEM Current Load is now: CRITICAL on mwreview-param3 i-000004dd.pmtpa.wmflabs output: Connection refused by host [16:54:32] PROBLEM Current Users is now: CRITICAL on mwreview-param3 i-000004dd.pmtpa.wmflabs output: Connection refused by host [16:55:13] PROBLEM Disk Space is now: CRITICAL on mwreview-param3 i-000004dd.pmtpa.wmflabs output: Connection refused by host [16:56:03] PROBLEM Free ram is now: CRITICAL on mwreview-param3 i-000004dd.pmtpa.wmflabs output: Connection refused by host [16:57:08] andrewbogott: What about "never use git if you don't want get crazy" ;-) [16:57:24] PROBLEM Total processes is now: CRITICAL on mwreview-param3 i-000004dd.pmtpa.wmflabs output: Connection refused by host [16:57:45] or never use Gerrit's "api" ;-( [16:58:53] RECOVERY Current Load is now: OK on mwreview-param3 i-000004dd.pmtpa.wmflabs output: OK - load average: 0.54, 0.94, 0.58 [16:59:33] RECOVERY Current Users is now: OK on mwreview-param3 i-000004dd.pmtpa.wmflabs output: USERS OK - 1 users currently logged in [17:00:12] RECOVERY Disk Space is now: OK on mwreview-param3 i-000004dd.pmtpa.wmflabs output: DISK OK [17:01:03] RECOVERY Free ram is now: OK on mwreview-param3 i-000004dd.pmtpa.wmflabs output: OK: 610% free memory [17:02:01] andrewbogott: The change is now in my opinion OK [17:02:15] cool; thanks for reading. I'm going to test it a bit more. [17:02:22] RECOVERY Total processes is now: OK on mwreview-param3 i-000004dd.pmtpa.wmflabs output: PROCS OK: 92 processes [17:04:46] Change on 12mediawiki a page Developer access was modified, changed by Sharihareswara (WMF) link https://www.mediawiki.org/w/index.php?diff=596743 edit summary: /* User:Prageck */ [17:10:42] PROBLEM Disk Space is now: WARNING on mw1-21beta-lucid i-00000416.pmtpa.wmflabs output: DISK WARNING - free space: / 78 MB (5% inode=51%): [17:14:45] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [17:15:57] [bz] (8NEW - created by: 2Antoine "hashar" Musso, priority: 4Unprioritized - 6normal) [Bug 41311] mwdeploy unix user should be in svn group - https://bugzilla.wikimedia.org/show_bug.cgi?id=41311 [17:44:47] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [18:01:47] andrewbogott: When I have a Puppetmaster, how can I select a class to install like in instance config [18:03:17] *I mean the labs-puppetmaster-role [18:04:34] Jan_Luca: I'm not sure if it's possible to do outside of the web interface. You can add a new arbitrary class to your project on this page https://labsconsole.wikimedia.org/wiki/Special:NovaPuppetGroup [18:04:37] Is that what you were asking? [18:05:39] Does this config work when the puppetmaster::self class is installed? [18:08:53] yep [18:17:02] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [18:24:33] RECOVERY Current Users is now: OK on parsoid-spof i-000004d6.pmtpa.wmflabs output: USERS OK - 0 users currently logged in [18:32:32] PROBLEM Current Users is now: WARNING on parsoid-spof i-000004d6.pmtpa.wmflabs output: USERS WARNING - 6 users currently logged in [18:43:52] PROBLEM Current Load is now: WARNING on parsoid-roundtrip5-8core i-000004db.pmtpa.wmflabs output: WARNING - load average: 6.26, 6.14, 5.51 [18:45:52] PROBLEM dpkg-check is now: CRITICAL on parsoid-roundtrip4 i-000004d7.pmtpa.wmflabs output: DPKG CRITICAL dpkg reports broken packages [18:47:02] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [18:55:52] RECOVERY dpkg-check is now: OK on parsoid-roundtrip4 i-000004d7.pmtpa.wmflabs output: All packages OK [19:07:33] PROBLEM Total processes is now: WARNING on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS WARNING: 196 processes [19:17:02] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [19:27:52] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: WARNING - load average: 3.96, 5.07, 5.04 [19:28:52] RECOVERY dpkg-check is now: OK on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: All packages OK [19:30:42] * andrewbogott back in 30 [19:32:31] petan: I owe you something :) [19:32:53] RECOVERY Current Load is now: OK on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: OK - load average: 0.67, 2.64, 4.03 [19:33:45] there's a problem with giving out admin access. you can modify the JS as much as you'd like [19:33:49] which means you could XSS the cloudadmins [19:34:00] well, not even XSS [19:34:02] you could just inject whatever you'd like [19:37:35] petan: I can give another group the "protect" right [19:37:35] I'll do that [19:38:55] RECOVERY Current Load is now: OK on parsoid-roundtrip5-8core i-000004db.pmtpa.wmflabs output: OK - load average: 2.18, 2.97, 4.70 [19:39:33] petan: I made a group called "contentadmin" and added you [19:42:29] Ryan_Lane (or jeremyb ) do you know if an ssh key is required for developer / gerrit access? (http://www.mediawiki.org/wiki/Commit_access) [19:42:41] you upload your own keys [19:42:45] ok [19:42:49] so, no [19:43:11] * aude wondering what the barrier to entry is for reviewing and commenting on gerrit [19:43:32] fairly low [19:43:41] ok [19:43:42] fill out http://www.mediawiki.org/wiki/Developer_access [19:43:45] that's it [19:43:57] soon you'll just need to self-register [19:43:57] yeah, but for the more ordinary wikipedians :) [19:44:17] maybe they are motivated enough to come to gerrit and comment on something, but not much more [19:45:11] it's the rare wikipedian but sure they exist [19:47:02] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [19:49:13] aude: yeah [19:49:31] petan: ok. I gave contentadmin a bunch of rights, including the ability to block users [19:49:47] should be all that's needed to manage content [19:49:58] it doesn't allow editing of site css/js, though [19:52:52] PROBLEM Free ram is now: CRITICAL on dumps-bot2 i-000003f4.pmtpa.wmflabs output: Critical: 5% free memory [20:01:53] PROBLEM Current Load is now: WARNING on parsoid-roundtrip5-8core i-000004db.pmtpa.wmflabs output: WARNING - load average: 9.18, 7.90, 5.95 [20:02:33] PROBLEM Total processes is now: CRITICAL on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS CRITICAL: 289 processes [20:07:55] PROBLEM Current Load is now: WARNING on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: WARNING - load average: 8.58, 8.05, 6.27 [20:10:53] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: WARNING - load average: 6.50, 6.81, 5.62 [20:17:05] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [20:17:35] PROBLEM Total processes is now: WARNING on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS WARNING: 196 processes [20:19:35] [bz] (8NEW - created by: 2Damian Z, priority: 4Highest - 6blocker) [Bug 41008] Clear out NFS - https://bugzilla.wikimedia.org/show_bug.cgi?id=41008 [20:19:52] RECOVERY Free ram is now: OK on dumps-bot1 i-000003ed.pmtpa.wmflabs output: OK: 33% free memory [20:40:13] how well does glusterfs normally cope with concurrency? [20:40:42] RECOVERY Disk Space is now: OK on mw1-21beta-lucid i-00000416.pmtpa.wmflabs output: DISK OK [20:40:56] we restarted a few clients that get their source from /data/project, and now the glusterfs process seems to eat away cpu without making progress [20:41:25] has been ongoing for a few minutes now [20:45:50] gwicke: what do you mean? [20:46:06] meaning it's not reading the files properly? [20:46:15] it should handle it as well as NFS [20:46:31] if not faster, but it seems it's slower, overall [20:46:45] the glusterfs process consumes ~140% cpu and the clients never start up [20:46:54] clients? [20:47:01] rt test clients [20:47:03] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [20:47:04] node.js processes [20:47:23] http://ganglia.wikimedia.org/latest/?r=hour&cs=&ce=&s=by+name&c=Glusterfs%2520cluster%2520pmtpa&tab=m&vn= [20:48:01] hm [20:48:12] where the hell are the virt nodes in ganglia? [20:48:12] are there any known issues around glusterfs and concurrent startuf of reading clients on a bunch of servers? [20:48:29] I don't believe so [20:48:43] PROBLEM Disk Space is now: WARNING on mw1-21beta-lucid i-00000416.pmtpa.wmflabs output: DISK WARNING - free space: / 78 MB (5% inode=51%): [20:49:02] ok [20:50:52] RECOVERY Current Load is now: OK on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: OK - load average: 1.52, 3.59, 4.89 [20:52:55] RECOVERY Current Load is now: OK on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: OK - load average: 0.65, 3.18, 4.95 [21:01:56] RECOVERY Current Load is now: OK on parsoid-roundtrip5-8core i-000004db.pmtpa.wmflabs output: OK - load average: 0.05, 1.48, 4.04 [21:01:56] gwicke: has gluster recovered? [21:01:58] http://ganglia.wikimedia.org/latest/?r=hour&cs=&ce=&s=by+name&c=Glusterfs%2520cluster%2520pmtpa&tab=m&vn= [21:02:10] something made it go a little insane for a bit [21:02:28] we stopped everything and are now investigating [21:03:14] Ryan_Lane: ^^ [21:03:21] ah [21:03:21] ok [21:08:06] regarding gluster issues i also filed https://bugzilla.wikimedia.org/show_bug.cgi?id=41123 some days ago just in case thats related [21:14:03] yeah, that may be unrelated [21:14:32] I'd surely hope there's not a lot of split brained files [21:14:35] that would suck [21:14:36] figured it out- the error was in our code, which caused quick restarts [21:14:43] ah, ok [21:14:44] sorry for the noise [21:14:45] that's good to hear [21:14:49] no worries [21:14:58] it's good to know if there's performance issues [21:15:07] Damianz: http://docs.saltstack.org/en/latest/topics/releases/0.10.4.html#external-authentication-system [21:16:29] Damianz: seems it's now possible to say "user x is allowed to run command y on minion z" [21:16:58] though this still puts a lot of trust onto the client [21:17:04] keystone auth would fix this [21:18:15] ugh [21:18:28] it seems this allows the user to run this from any minion too [21:18:56] it would be nice if it was "user x is allowed to run command y on minion z from minion a" [21:19:12] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [21:19:33] ah [21:19:39] it does have this, in the peer system [21:19:45] \o/ [21:24:56] PROBLEM Current Load is now: WARNING on parsoid-roundtrip5-8core i-000004db.pmtpa.wmflabs output: WARNING - load average: 13.25, 10.76, 6.47 [21:25:54] PROBLEM Current Load is now: WARNING on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: WARNING - load average: 10.42, 9.40, 6.36 [21:33:52] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: WARNING - load average: 8.04, 8.02, 6.12 [21:34:52] PROBLEM Current Load is now: WARNING on parsoid-roundtrip4 i-000004d7.pmtpa.wmflabs output: WARNING - load average: 5.58, 5.81, 5.14 [21:42:04] nagios notices that rt testing is back up to speed ;) [21:49:13] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [22:02:34] PROBLEM Total processes is now: CRITICAL on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS CRITICAL: 289 processes [22:12:32] PROBLEM Total processes is now: WARNING on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS WARNING: 196 processes [22:19:13] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [22:39:52] RECOVERY Current Load is now: OK on parsoid-roundtrip4 i-000004d7.pmtpa.wmflabs output: OK - load average: 3.89, 3.64, 4.77 [22:43:39] !log bugtracker adding new member aklapper (plus sysadmin) [22:43:41] Logged the message, Master [22:44:07] andre__: https://labsconsole.wikimedia.org/wiki/Nova_Resource:Bugtracker [22:44:33] yepp [22:44:49] andre__: well, i dont see any instances in that project [22:49:12] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [22:52:33] RECOVERY Total processes is now: OK on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS OK: 102 processes [22:54:12] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [23:03:53] RECOVERY Current Load is now: OK on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: OK - load average: 2.70, 3.36, 4.44 [23:19:12] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [23:24:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs CRITICAL - Plugin timed out after 10 seconds [23:40:53] RECOVERY Current Load is now: OK on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: OK - load average: 4.32, 4.52, 4.96 [23:43:33] PROBLEM Total processes is now: WARNING on aggregator-test1 i-000002bf.pmtpa.wmflabs output: PROCS WARNING: 151 processes [23:44:53] RECOVERY Current Load is now: OK on parsoid-roundtrip5-8core i-000004db.pmtpa.wmflabs output: OK - load average: 4.37, 4.41, 4.88 [23:46:02] 10/23/2012 - 23:46:00 - User swalling may have been modified in LDAP or locally, updating key in project(s): editor-engagement [23:46:17] 10/23/2012 - 23:46:16 - Updating keys for swalling at /export/keys/swalling [23:48:32] RECOVERY Total processes is now: OK on aggregator-test1 i-000002bf.pmtpa.wmflabs output: PROCS OK: 146 processes [23:49:13] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [23:55:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100%