[00:52:14] PROBLEM Free ram is now: WARNING on bots-sql3 bots-sql3 output: Warning: 19% free memory [01:02:14] RECOVERY Free ram is now: OK on bots-sql3 bots-sql3 output: OK: 20% free memory [01:20:14] PROBLEM Free ram is now: WARNING on bots-sql3 bots-sql3 output: Warning: 19% free memory [02:40:14] RECOVERY Free ram is now: OK on bots-sql3 bots-sql3 output: OK: 21% free memory [02:42:54] PROBLEM Free ram is now: WARNING on puppet-lucid puppet-lucid output: Warning: 12% free memory [02:52:54] PROBLEM Free ram is now: CRITICAL on puppet-lucid puppet-lucid output: Critical: 3% free memory [03:08:14] PROBLEM Free ram is now: WARNING on bots-sql3 bots-sql3 output: Warning: 19% free memory [03:27:54] RECOVERY Free ram is now: OK on puppet-lucid puppet-lucid output: OK: 20% free memory [13:36:44] PROBLEM Free ram is now: WARNING on deployment-backup deployment-backup output: Warning: 19% free memory [13:46:44] PROBLEM host: deployment-web is DOWN address: deployment-web CRITICAL - Host Unreachable (deployment-web) [13:47:54] RECOVERY host: deployment-web is UP address: deployment-web PING OK - Packet loss = 0%, RTA = 0.82 ms [15:50:17] when will I want a labs account generally? [15:50:29] I still fail to understand what it can provide.. [15:50:44] liangent: there have been some updates to https://www.mediawiki.org/wiki/WMF_Projects/Wikimedia_Labs [15:51:13] sumanah: I've seen that page [15:51:35] liangent: are you happy hosting your own MediaWiki development environment? [15:51:40] if you are, then you don't need Labs. [15:52:37] Well, Labs also supports volunteer contributions to WMF's operational work [15:53:01] liangent: are you interested in systems administration? [15:53:04] sumanah: a simple one is good but there was time when I want to fix a bug but I didn't do so because I don't want to set up the environment [15:53:06] I didn't think you were, but I could be wrong [15:53:16] liangent: in that case, a labs instance would benefit you [15:53:24] RoanKattouw: eg Bug 34272 I submitted just now [15:53:37] Ah, right [15:53:56] Yeah, in The Future, you'll be able to say "give me a labs instance that's set up to run MW" and you'll just get one [15:54:16] Right now, producing instances with MW installed and ready to go hasn't been done yet [15:54:32] liangent: so, Ryan will alert the development community when that sort of functionality is available [15:54:38] liangent: until then, you probably won't want one. [15:56:19] in labs can I get the same environment as if the code is run on production servers? [15:56:25] eg. multiple layers of caching [15:57:01] eventually, yes! [15:57:12] but now, no? [15:57:20] Yes, we are building a complete clone of the production cluster in labs [15:57:25] Emphasis on "are building" :) [15:57:38] That's right, the clone isn't ready yet [15:57:47] The Squid config isn't in there at all, for instance [15:57:55] hmm [15:58:21] will there be a better db access? [15:59:16] liangent: it sounds like we need Ryan to update https://www.mediawiki.org/wiki/Wikimedia_Labs#Roadmap with some rough dates [16:00:15] Access to the production DB or some mock version thereof is also something that's being worked on [16:03:28] liangent: ok, I've requested on https://www.mediawiki.org/wiki/Talk:Wikimedia_Labs [16:05:49] sumanah: so long your signature is [16:06:01] liangent: yeah, maybe I should shorten it. [16:08:30] is there such a feature on labs that I can restore a project to a previous state saved before [16:11:38] What do you mean exactly? [16:13:06] !g [16:13:06] https://gerrit.wikimedia.org/r/$1 [16:13:27] RoanKattouw: eg when I finished some work I want to get back to a clean mw installation [16:14:10] You could just create a new VM, and optionally delete the old one [16:14:43] but if I delete it I have to reinstall/configure mediawiki itself I think [16:15:31] or is there currently an option to clone an existing project [16:15:38] Well ideally it'd be installed for you [16:15:45] I don't think there is [16:15:51] That would be a nice feature request I guess [16:17:15] should it be logged somewhere..? [16:17:31] by the way is it another use of labs to run bots? [16:17:47] and what's the difference between labs and ts on this side [16:19:52] I believe you'll be able to run bots and that sort of thing on labs, so in that sense it wouldn't be too different from the toolserver [16:20:08] But those are excellent questions for Ryan Lane, who is on vacation now [16:23:33] It looks to me like cloning is supported in openstack, so it might be possible to implement that in labs at some point... [16:25:45] The 'one true way' to do what you want, though, is to set up a puppet configuration that will create a perfectly-configured VM out of thin air. [16:28:59] petan|wk: any idea why gadgets aren't working on enwiki? [16:34:55] RoanKattouw: You can't take a vacation from fun open projects! [16:35:02] hehe [16:36:21] methecooldude: Do you still have that eggdrop twitter stuff by any chance? [17:03:15] 02/08/2012 - 17:03:15 - Creating a home directory for catrope at /export/home/deployment-prep/catrope [17:04:16] 02/08/2012 - 17:04:15 - Updating keys for catrope [17:09:59] !log deployment-prep Added self to depops group using sudo [17:10:12] !log deployment-prep ...on deployment-web [17:11:03] WTF [17:11:09] Why can't I write to this file [17:11:36] RoanKattouw: nfs mount [17:11:44] right [17:11:46] try to do it on nfs-memc [17:11:49] I need to edit it on ... yeah [17:13:14] !log and on deployment-nfs-memc too [17:13:41] !log deployment-prep and on deployment-nfs-memc too [17:13:49] !log deployment-prep Enabled $wgResourceLoaderExperimentalAsyncLoading in CommonSettingsDeploy.php [17:15:33] !log deployment-prep Didn't fix the problem, reverting [18:08:46] hexmode: Could you make me a sysop on commons beta? [18:09:14] RoanKattouw: sure, 1s [18:10:49] http://commons.wikimedia.beta.wmflabs.org/wiki/Special:UserRights/catrope [18:12:05] !log deployment-prep svn up & update data for Bug #34245 [18:12:33] LOL, thanks [18:27:11] Damianz: Maybe... dunno [18:41:25] * RoanKattouw WTFs, can't reproduce the RL issue after adding debugging code [18:47:47] methecooldude: I hacked something up in node so it dm :) [18:48:08] RoanKattouw: WTFs are those unit tests that fail one in every 200thousand. [18:49:31] methecooldude: Though as you're here - have you seen any moans about cbng feeds recently? More specifically after I replaced the bots. [18:57:13] petan, what hour is planned that labs conf ? [19:19:58] !log deployment-prep Recursively setting everything in /mnt/export/apache/common/live group-writeable [19:26:45] :o [19:49:14] RECOVERY Free ram is now: OK on bots-2 bots-2 output: OK: 25% free memory [21:38:27] !log deployment-prep disable aftv5 for enwiki -- it was breaking gadgets [21:38:37] \o/ [21:38:46] that one drove me crazy [21:40:56] :O [21:40:58] how? [22:18:31] RoanKattouw: tracking it down took too long. no js errors, just no loading gadgets and last bit of js to load was (after searching) aftv5 [22:19:15] Strange [22:34:03] hexmode: do Dario and Fabrice know about this? (regarding aftv5) [22:34:04] PROBLEM Total Processes is now: CRITICAL on nova-test1 nova-test1 output: Connection refused by host [22:34:27] drdee: I filed a bug, you want the num? [22:34:34] PROBLEM dpkg-check is now: CRITICAL on nova-test1 nova-test1 output: CHECK_NRPE: Error - Could not complete SSL handshake. [22:34:38] I find this very strange though [22:34:55] I assume things magically started working when you disabled AFTv5 [22:35:02] Do they magically stop working again once you reenable AFTv5? [22:35:11] I didn't try [22:35:15] shall I? [22:35:24] * hexmode goes to poke [22:35:54] PROBLEM Current Load is now: CRITICAL on nova-test1 nova-test1 output: CHECK_NRPE: Error - Could not complete SSL handshake. [22:36:24] PROBLEM Current Users is now: CRITICAL on nova-test1 nova-test1 output: CHECK_NRPE: Error - Could not complete SSL handshake. [22:37:02] RoanKattouw: appears to still work [22:37:04] PROBLEM Disk Space is now: CRITICAL on nova-test1 nova-test1 output: CHECK_NRPE: Error - Could not complete SSL handshake. [22:37:12] let me make sure aft is loading [22:37:44] PROBLEM Free ram is now: CRITICAL on nova-test1 nova-test1 output: CHECK_NRPE: Error - Could not complete SSL handshake. [22:38:10] hexmode: please [22:38:59] (about giving me the bug id) [22:39:16] drdee: https://bugzilla.wikimedia.org/show_bug.cgi?id=34283 [22:42:37] RoanKattouw: I've re-enabled, but now I don't see AFTv5? [22:42:44] * hexmode is confused [22:43:00] Oh, hmm [22:43:05] Try touching startup.js [22:43:09] And hard refreshing [22:48:42] how do I see aft on an article? [22:49:09] but, assuming it is there, I turned it back on and gadgets are still working [22:49:39] Put it in Category:Article_Feedback_5 [22:49:55] the test article that is [22:52:25] yes, looks like a rl or cache problem [22:52:37] http://en.wikipedia.beta.wmflabs.org/wiki/Hurricane_Celeste_%281972%29 [22:52:47] gadgets and aftv5 [22:57:04] OK, good [22:57:14] andrewbogott: do you know how I'd grant permissions to myself on my labs instance? [22:57:17] Then it's just the categorytree bug, right? [22:57:31] nimish_g: Does sudo not work for you? [22:57:39] nope [22:57:52] RoanKattouw: ^^ [22:58:20] Which project is this? [22:58:33] You are a sysadmin+netadmin for every project you're a member of, except bastion (of course) [22:58:35] the project is called "outreach" [22:58:40] yep [22:59:19] You should just have sudo on those instances [23:00:11] it's asking me for a password...I don't know that I have a password set anywhere [23:00:21] Oh [23:00:26] That's just your labsconsole password [23:00:33] oooh ok [23:00:37] there we go. thanks!! [23:00:43] Which is linked to your wiki username which is not necessarily your shell username [23:01:06] right, that's what made it confusing [23:01:25] It's a bit messed up. In theory, your shell username plus SSH key are used for anything that's shell-based (SSH, SVN, git) and your wiki username plus password are used for anything that's web-based (labsconsole, Gerrit) [23:01:35] But sudo is the one exception apparently [23:04:21] I guess that makes sense in some weird way or another