[01:35:59] PROBLEM Puppet freshness is now: CRITICAL on incubator-web i-00000198 output: Puppet has not run in the last 10 hours [01:35:59] PROBLEM Puppet freshness is now: CRITICAL on kripke i-000001aa output: Puppet has not run in the last 10 hours [01:35:59] PROBLEM Puppet freshness is now: CRITICAL on labs-lvs1 i-00000057 output: Puppet has not run in the last 10 hours [01:35:59] PROBLEM Puppet freshness is now: CRITICAL on nova-daas-1 i-000000e7 output: Puppet has not run in the last 10 hours [01:35:59] PROBLEM Puppet freshness is now: CRITICAL on orgcharts-dev i-0000018f output: Puppet has not run in the last 10 hours [01:35:59] PROBLEM Puppet freshness is now: CRITICAL on pad1 i-00000066 output: Puppet has not run in the last 10 hours [01:35:59] PROBLEM Puppet freshness is now: CRITICAL on patchtest2 i-000000fd output: Puppet has not run in the last 10 hours [01:36:00] PROBLEM Puppet freshness is now: CRITICAL on pediapress-puppetmaster i-000001af output: Puppet has not run in the last 10 hours [01:36:00] PROBLEM Puppet freshness is now: CRITICAL on puppet-lucid i-00000080 output: Puppet has not run in the last 10 hours [01:36:01] PROBLEM Puppet freshness is now: CRITICAL on reportcard1 i-000000a8 output: Puppet has not run in the last 10 hours [01:36:01] PROBLEM Puppet freshness is now: CRITICAL on search-test i-000000cb output: Puppet has not run in the last 10 hours [01:36:02] PROBLEM Puppet freshness is now: CRITICAL on test-oneiric i-00000187 output: Puppet has not run in the last 10 hours [03:40:31] PROBLEM Free ram is now: WARNING on utils-abogott i-00000131 output: Warning: 14% free memory [03:44:11] PROBLEM Free ram is now: WARNING on test-oneiric i-00000187 output: Warning: 14% free memory [03:49:11] PROBLEM Free ram is now: WARNING on orgcharts-dev i-0000018f output: Warning: 15% free memory [03:49:11] PROBLEM Free ram is now: WARNING on nova-daas-1 i-000000e7 output: Warning: 15% free memory [03:55:31] PROBLEM Free ram is now: CRITICAL on utils-abogott i-00000131 output: Critical: 5% free memory [03:59:11] PROBLEM Free ram is now: CRITICAL on test-oneiric i-00000187 output: Critical: 5% free memory [04:00:31] RECOVERY Free ram is now: OK on utils-abogott i-00000131 output: OK: 97% free memory [04:04:11] RECOVERY Free ram is now: OK on test-oneiric i-00000187 output: OK: 97% free memory [04:09:11] PROBLEM Free ram is now: CRITICAL on orgcharts-dev i-0000018f output: Critical: 4% free memory [04:14:11] RECOVERY Free ram is now: OK on orgcharts-dev i-0000018f output: OK: 96% free memory [04:14:11] PROBLEM Free ram is now: CRITICAL on nova-daas-1 i-000000e7 output: Critical: 4% free memory [04:19:11] RECOVERY Free ram is now: OK on nova-daas-1 i-000000e7 output: OK: 93% free memory [04:21:01] PROBLEM Free ram is now: WARNING on test3 i-00000093 output: Warning: 7% free memory [04:26:01] RECOVERY Free ram is now: OK on test3 i-00000093 output: OK: 96% free memory [06:01:21] PROBLEM Puppet freshness is now: CRITICAL on shop-analytics-main1 i-000001d0 output: Puppet has not run in the last 10 hours [06:44:18] PROBLEM Current Load is now: WARNING on bots-cb i-0000009e output: WARNING - load average: 3.91, 15.86, 9.50 [06:45:30] PROBLEM Current Load is now: CRITICAL on nagios 127.0.0.1 output: CRITICAL - load average: 3.65, 8.71, 5.34 [06:51:57] PROBLEM Current Load is now: WARNING on bots-sql3 i-000000b4 output: WARNING - load average: 5.50, 7.41, 5.51 [06:55:27] PROBLEM Current Load is now: WARNING on nagios 127.0.0.1 output: WARNING - load average: 0.26, 2.05, 3.47 [06:56:57] RECOVERY Current Load is now: OK on bots-sql3 i-000000b4 output: OK - load average: 0.63, 3.58, 4.42 [06:59:17] RECOVERY Current Load is now: OK on bots-cb i-0000009e output: OK - load average: 0.34, 1.39, 4.10 [07:00:27] RECOVERY Current Load is now: OK on nagios 127.0.0.1 output: OK - load average: 0.29, 1.00, 2.61 [07:29:05] join #wikichem [07:37:54] wikichem :O [07:38:06] that's something new [07:43:44] PROBLEM Current Load is now: CRITICAL on hugglewa-1 i-000001dc output: Connection refused by host [07:44:24] PROBLEM Current Users is now: CRITICAL on hugglewa-1 i-000001dc output: Connection refused by host [07:45:04] PROBLEM Disk Space is now: CRITICAL on hugglewa-1 i-000001dc output: Connection refused by host [07:45:44] PROBLEM Free ram is now: CRITICAL on hugglewa-1 i-000001dc output: Connection refused by host [07:46:54] PROBLEM Total Processes is now: CRITICAL on hugglewa-1 i-000001dc output: Connection refused by host [07:47:34] PROBLEM dpkg-check is now: CRITICAL on hugglewa-1 i-000001dc output: Connection refused by host [08:01:16] wtf is bz-dev [10:40:40] andrewbogott: hi [10:40:54] is deployment ok now for Erik and so [10:41:06] I didn't receive update in email for some time [10:41:23] I see that wizard doesn't work but that's probably a bug in that [11:27:13] PROBLEM Free ram is now: CRITICAL on bots-3 i-000000e5 output: CHECK_NRPE: Socket timeout after 10 seconds. [11:31:49] PROBLEM Free ram is now: WARNING on bots-3 i-000000e5 output: Warning: 16% free memory [11:36:09] PROBLEM Puppet freshness is now: CRITICAL on incubator-web i-00000198 output: Puppet has not run in the last 10 hours [11:36:09] PROBLEM Puppet freshness is now: CRITICAL on kripke i-000001aa output: Puppet has not run in the last 10 hours [11:36:09] PROBLEM Puppet freshness is now: CRITICAL on labs-lvs1 i-00000057 output: Puppet has not run in the last 10 hours [11:36:09] PROBLEM Puppet freshness is now: CRITICAL on nova-daas-1 i-000000e7 output: Puppet has not run in the last 10 hours [11:36:09] PROBLEM Puppet freshness is now: CRITICAL on orgcharts-dev i-0000018f output: Puppet has not run in the last 10 hours [11:36:09] PROBLEM Puppet freshness is now: CRITICAL on pad1 i-00000066 output: Puppet has not run in the last 10 hours [11:36:09] PROBLEM Puppet freshness is now: CRITICAL on patchtest2 i-000000fd output: Puppet has not run in the last 10 hours [11:36:10] PROBLEM Puppet freshness is now: CRITICAL on pediapress-puppetmaster i-000001af output: Puppet has not run in the last 10 hours [11:36:10] PROBLEM Puppet freshness is now: CRITICAL on puppet-lucid i-00000080 output: Puppet has not run in the last 10 hours [11:36:11] PROBLEM Puppet freshness is now: CRITICAL on reportcard1 i-000000a8 output: Puppet has not run in the last 10 hours [11:36:11] PROBLEM Puppet freshness is now: CRITICAL on search-test i-000000cb output: Puppet has not run in the last 10 hours [11:36:12] PROBLEM Puppet freshness is now: CRITICAL on test-oneiric i-00000187 output: Puppet has not run in the last 10 hours [12:45:08] PROBLEM Current Load is now: CRITICAL on nagios 127.0.0.1 output: CRITICAL - load average: 2.92, 6.54, 3.59 [12:50:08] RECOVERY Current Load is now: OK on nagios 127.0.0.1 output: OK - load average: 0.20, 2.70, 2.73 [13:15:51] petan: I think that performance is still not great. Ryan suggested that the cluster could use some caching work and/or bigger instances... I'll forward you his email. [13:26:38] We need a proxy infront of the squid instances then a bunch more squids and some caching re-work IIRC [14:38:23] andrewbogott: what is the desired speed of cluster? :) [14:38:44] there is already squid, which ins't well configured because I still wait for Ryan [14:39:02] however pages loads to me within 5 seconds, almost as fast as on prod [14:39:06] petan: I don't think there's an absolute requirement, other than 'fast enough that the developers stop complaining' :) [14:39:27] ok, maybe we need more info about the performance problems then. Could be that it's intermittent or something. [14:40:05] but, it definitely is the case that sometimes deployment-web* systems die due to lack of memory. So rebuilding them on bigger instances seems like a no-brainer. [14:40:20] if there is a space for that [14:40:28] I thought Ryan was like we are out of ram [14:40:43] so all new servers have max 2 gb [14:41:03] if it's ok I can create 2 extra servers with 4gb of ram [14:41:19] but right now we have 6 servers and they seems to be stable [14:41:42] so... [14:42:59] Yeah, I'm not clear what's happening with labs capacity. I predict that Ryan will appear online in a few hours, and we can sort this out with his help. [14:43:50] I think it's problem with configuration rather than number of servers [14:44:27] Last week web5 was crashing every day or so. [14:45:02] web5 is 1 gb [14:45:13] if it crash again I make it 4 [14:46:01] Take note that I don't know /why/ it crashed. Ryan thought it was out of memory but I didn't verify. [14:46:27] ok [15:31:21] Change abandoned: Demon; "We don't want this unmaintained junk." [operations/puppet] (test) - https://gerrit.wikimedia.org/r/3521 [15:31:53] wut [15:42:55] http://www.youtube.com/social/blog/2lnbO9ly0Rc < LOL [16:01:38] PROBLEM Puppet freshness is now: CRITICAL on shop-analytics-main1 i-000001d0 output: Puppet has not run in the last 10 hours [17:40:38] PROBLEM Puppet freshness is now: CRITICAL on hugglewa-1 i-000001dc output: Puppet has not run in the last 10 hours [18:54:48] PROBLEM Current Load is now: WARNING on deployment-web5 i-00000176 output: WARNING - load average: 1.20, 12.45, 8.36 [19:04:48] RECOVERY Current Load is now: OK on deployment-web5 i-00000176 output: OK - load average: 0.07, 1.73, 4.40 [19:20:07] Ryan_Lane: Which host did you suggest I use for generating random test pages via the wm api? [19:20:11] (welcome back, btw) [19:29:11] Ryan_Lane: http://i.imgur.com/2RM1U.gif [20:02:38] andrewbogott: thanks [20:03:41] andrewbogott: hm [20:03:51] andrewbogott: probably beta [20:04:10] andrewbogott: or any mediawiki instance, really [20:04:24] I have one on labs-production1 [20:04:48] though it's not terribly easy to use, since it's set up for ldap auth and such [20:19:02] <^demon> Ryan_Lane: Welcome back :) [20:19:08] thanks [20:20:35] <^demon> I know you've probably got 8000 things people are asking you to do, but I've got some changes to the gerrit hooks I've submitted and would love review when you get a chance. [20:21:00] did you add me as a reviewer? [20:21:05] <^demon> Yep [20:25:18] <^demon> I was rather proud of myself. It's the most python I've ever written ;-) [20:25:33] :D [20:25:37] I <3 python [20:25:52] <^demon> I don't mind it as much as I did. [20:28:05] <^demon> I also opened a bug based on some complaints I heard last week about port 29418 :p [20:28:23] heh [20:28:26] it's not that much python :) [20:28:49] did you test this? [20:28:59] <^demon> No, I hadn't yet [20:29:04] also, was this user created? [20:29:12] <^demon> Was what user created? [20:29:22] l10n-bot [20:29:51] <^demon> l10n-bot is the dedicated user we setup last week for translatewiki. [20:30:10] <^demon> Only thing that commits as him is their export scripts. [20:30:17] did you run a lint check on this? :) [20:30:40] <^demon> I'm a total python noob, so assume the worst. [20:30:41] * Ryan_Lane already knows you didn't ;) [20:31:18] <^demon> What needs fixing? I can amend. [20:31:30] I'm adding a note [20:31:38] <^demon> mmk [20:32:23] gerrit review? [20:32:28] does that work? [20:32:34] <^demon> Yes [20:32:42] <^demon> `gerrit review` is the new name. [20:32:42] in this version? [20:32:45] <^demon> Yes [20:32:51] oh. cool [20:33:21] the new name for what? [20:33:30] <^demon> Command line review interface. [20:33:33] <^demon> `gerrit approve` is technically deprecated. [20:33:40] <^demon> So I chose the new name so we don't get bit later. [20:33:52] ah... I didn't even know there was a commandline. [20:34:15] <^demon> https://gerrit.wikimedia.org/r/Documentation/cmd-index.html - there's docs :) [20:36:59] <^demon> Ryan_Lane: Anything else I should fix, or is it otherwise ok? [20:37:09] did you look at the review? [20:37:14] oh [20:37:17] *else*. heh [20:37:54] <^demon> Yeah I've got that fix staged :) [20:39:04] * Damianz wonders if Ryan_Lane has 2 heads [20:41:32] well, I'm a Gemini [20:42:52] * ^demon takes Ryan's silence to mean assent. [20:43:28] heh [20:43:31] nothing else that I see [20:43:39] it would be nice if you could test this first, though [20:43:46] but meh, I'll just merge it [20:43:57] after you fix it [20:44:20] * Ryan_Lane sighs [20:44:44] I have a ton of fixes for OpenStackManager, but there's been a bunch of mostly minor changes in svn [20:44:56] I'm going to have a ton of merge conflicts [20:45:11] ^demon: did you move my extensions to git yet? :) [20:45:21] <^demon> No, I'll do it with the batch I'm doing Friday [20:45:25] ok [20:45:43] my changes should make "Manage addresses" far less slow [20:46:04] also, it adds a project filter, so if you are in 20 projects, you can select only the ones you want to see right now [20:46:13] the default filter is no projects, though [20:46:44] is there any way for me to change the layout of the checkboxes in HTMLForm? [20:46:55] one long list of checkboxes is going to really suck for 100+ projects [20:46:57] <^demon> No clue. [20:47:15] eventually I'll need to modify that too [20:47:37] to be a paged selection, or an autocomplete textbox [20:47:57] it would be awesome to have an autocomplete checkbox list :) [20:49:00] oh. someone wanted to work on OpenStackManager for GSOC [20:49:04] I need to make them an instance to work in [20:50:27] Just give them root and vim :D [20:50:35] <^demon> I hope someone works on gerrit for gsoc :) [20:50:57] Damianz: I need to give them a working nova clone [20:51:08] with ldap and mediawiki, etc [20:51:38] Ryan_Lane: you should puppetize mediawiki while you're at it :) [20:51:47] would be nice [20:52:13] <^demon> MediaWiki doesn't want to be tied down like that ;-) [20:52:27] probably wouldn't be amazingly hard, depending on how I did it [20:53:03] <^demon> Command line installer might need some minor tweaks, but most of the legwork is done in that area. [20:54:01] bah [20:54:05] fucking brewster [20:54:19] its disk filled earlier so squid and lighttpd died [20:54:24] so instance creation is broken [20:56:12] lol [20:56:19] Can we get brewster a sister? [20:56:34] <^demon> Ugh, puns. [21:05:43] Damianz: maybe a punky one? [21:06:29] Really need food =/ [21:10:17] <^demon> Ryan_Lane: I'm adding you to the list of extensions I'm migrating. It's just OpenStackManager and LdapAuthentication, right? [21:10:32] for now, yeah [21:10:40] it's the only ones I'm actively maintaining [21:11:55] <^demon> Okie dokie [21:13:56] ^demon: did you amend that change? [21:14:24] PROBLEM Current Load is now: CRITICAL on nova-gsoc1 i-000001de output: CHECK_NRPE: Error - Could not complete SSL handshake. [21:14:56] <^demon> Yep [21:15:04] PROBLEM Current Users is now: CRITICAL on nova-gsoc1 i-000001de output: CHECK_NRPE: Error - Could not complete SSL handshake. [21:15:44] PROBLEM Disk Space is now: CRITICAL on nova-gsoc1 i-000001de output: CHECK_NRPE: Error - Could not complete SSL handshake. [21:16:14] PROBLEM Free ram is now: CRITICAL on nova-gsoc1 i-000001de output: CHECK_NRPE: Error - Could not complete SSL handshake. [21:17:34] PROBLEM Total Processes is now: CRITICAL on nova-gsoc1 i-000001de output: CHECK_NRPE: Error - Could not complete SSL handshake. [21:18:14] PROBLEM dpkg-check is now: CRITICAL on nova-gsoc1 i-000001de output: CHECK_NRPE: Error - Could not complete SSL handshake. [21:21:55] petr: Any suggestions about how I can create/get an account on en.wikipedia.beta.wmflabs.org? I get errors if I try to create one via the GUI [21:27:14] andrewbogott: oh? it should be working [21:28:04] I got an error when creating an account and now get the same error when trying to log in with that account. [21:28:16] It's the normal db-error-did-you-run-the-update-script page. [21:36:45] Thehelpfulone: working for you? [21:36:54] well I had my account created already [21:36:55] let me see [21:37:14] PROBLEM Puppet freshness is now: CRITICAL on incubator-web i-00000198 output: Puppet has not run in the last 10 hours [21:37:14] PROBLEM Puppet freshness is now: CRITICAL on kripke i-000001aa output: Puppet has not run in the last 10 hours [21:37:14] PROBLEM Puppet freshness is now: CRITICAL on labs-lvs1 i-00000057 output: Puppet has not run in the last 10 hours [21:37:14] PROBLEM Puppet freshness is now: CRITICAL on nova-daas-1 i-000000e7 output: Puppet has not run in the last 10 hours [21:37:14] PROBLEM Puppet freshness is now: CRITICAL on orgcharts-dev i-0000018f output: Puppet has not run in the last 10 hours [21:37:14] PROBLEM Puppet freshness is now: CRITICAL on pad1 i-00000066 output: Puppet has not run in the last 10 hours [21:37:14] PROBLEM Puppet freshness is now: CRITICAL on patchtest2 i-000000fd output: Puppet has not run in the last 10 hours [21:37:15] PROBLEM Puppet freshness is now: CRITICAL on pediapress-puppetmaster i-000001af output: Puppet has not run in the last 10 hours [21:37:15] PROBLEM Puppet freshness is now: CRITICAL on puppet-lucid i-00000080 output: Puppet has not run in the last 10 hours [21:37:16] PROBLEM Puppet freshness is now: CRITICAL on reportcard1 i-000000a8 output: Puppet has not run in the last 10 hours [21:37:16] PROBLEM Puppet freshness is now: CRITICAL on search-test i-000000cb output: Puppet has not run in the last 10 hours [21:37:17] PROBLEM Puppet freshness is now: CRITICAL on test-oneiric i-00000187 output: Puppet has not run in the last 10 hours [21:37:24] so I'm logged in andrewbogott [21:37:34] but that could just because I've been logged in for a while [21:37:36] * Thehelpfulone logs out [21:37:51] Well, I suspect that my login problem is caused by my account-creation problem. [21:40:19] * Damianz uses gerrit-wm to smash useless stuff up with [22:23:33] 04/02/2012 - 22:23:33 - Creating a project directory for packaging [22:23:33] 04/02/2012 - 22:23:33 - Creating a home directory for diederik at /export/home/packaging/diederik [22:23:34] 04/02/2012 - 22:23:33 - Creating a home directory for ben at /export/home/packaging/ben [22:24:34] 04/02/2012 - 22:24:33 - Updating keys for diederik [22:24:34] 04/02/2012 - 22:24:33 - Updating keys for ben [22:34:23] PROBLEM Current Load is now: CRITICAL on udp-filter i-000001df output: Connection refused by host [22:35:03] PROBLEM Current Users is now: CRITICAL on udp-filter i-000001df output: Connection refused by host [22:35:43] PROBLEM Disk Space is now: CRITICAL on udp-filter i-000001df output: Connection refused by host [22:36:13] PROBLEM Free ram is now: CRITICAL on udp-filter i-000001df output: Connection refused by host [22:37:13] hey ryan: i am having issues connecting to my new instance udp-filter in the package project? do you know what's going on? [22:37:30] did it finish building/ [22:37:33] PROBLEM Total Processes is now: CRITICAL on udp-filter i-000001df output: Connection refused by host [22:37:34] did it fail to build? [22:37:42] look at its console output [22:37:51] it looks good, i think [22:38:13] PROBLEM dpkg-check is now: CRITICAL on udp-filter i-000001df output: Connection refused by host [22:39:12] nagios is also issuing warnings [22:39:27] that's normal on new instances [22:40:23] nscd seems to not have started [22:40:26] it should work now [22:40:33] thx! [22:40:36] yw [22:40:54] it would have started working on next puppet run [22:42:28] welcome back Ryan_Lane :-D [22:42:33] hope you enjoyed your trip [22:42:38] I did :) [22:43:37] if you wanna do some python, I have hacked up something to have the puppet linter to output its error as monospaced [22:43:39] https://gerrit.wikimedia.org/r/#change,4090 [22:43:47] which "might" work :-D [22:44:02] I am not sure how to reliably test it [22:45:50] hahaha [22:46:04] I like the "might" part [22:46:19] the output in gerrit is confusing [22:46:30] shouldn't that be postfix? [22:46:33] it seems the multiline error message is made a oneliner [22:46:35] r [22:46:45] oh [22:46:48] no. it's prefix [22:47:10] you made it so puppet set "verify" for us ;-) [22:47:17] what if the message has newlines? [22:47:41] \n become \n [22:47:54] so it will still have newline, just that any error will end up monospaced [22:48:03] well, let's try it :) [22:49:17] ok. wtf [22:49:19] I can't merge? [22:49:40] conspiracy!!! [22:50:41] bah [22:50:46] you didn't rebase? [22:50:51] it depends on something else [22:50:59] oh you are right wrong dep [22:51:01] let me fix it [22:51:39] I think Gerrit 2.3 has a rebase button [22:52:02] https://gerrit.wikimedia.org/r/#change,4090 <-- rebased with patchset 2 !! [22:52:03] that would be *sweet* [22:52:27] there we go [22:52:52] now let me push in a failed puppet lint check [22:53:19] hi ryan: how can i clone from git from within a labs instance? [22:53:31] anonymously [22:53:43] or by forwarding your key to the instance, which is a bad idea [22:53:50] gerrit.wikimedia.org[0: 208.80.154.152]: errno=Connection refused [22:53:50] fatal: unable to connect a socket (Connection refused) [22:54:02] for port 443? [22:54:14] using https? [22:54:27] or, are you using ssh and forgetting to use port 29418? [22:54:42] rebase button commit is http://code.google.com/p/gerrit/source/detail?r=06e41266a213c5d0b43382ce85721b7cd16d670a [22:55:21] now trying https [22:55:38] error: Failed connect to gerrit.wikimedia.org:29416; Operation now in progress while accessing https://gerrit.wikimedia.org:29416/analytics/udp-filters.git/info/refs [22:56:32] ummm [22:56:32] no [22:56:36] with https don't provide a port [22:56:40] https is port 443 [22:56:43] 29418 is for ssh [22:57:02] without port: fatal: https://gerrit.wikimedia.org/analytics/udp-filters.git/info/refs not found: did you run git update-server-info on the server? [22:57:23] does that repo exist? [22:57:39] oh. it may be a different url with https [22:58:04] New patchset: Hashar; "invalid puppet change" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/4144 [22:58:14] New review: gerrit2; "Change did not pass lint check. You will need to send an amended patchset for this (see: https://lab..." [operations/puppet] (test); V: -1 - https://gerrit.wikimedia.org/r/4144 [22:58:19] hasharCall: heh. I had just checked it ;0 [22:58:28] oh [22:58:47] diederik: https://gerrit.wikimedia.org/r/p/udp-filters.git [22:58:57] err [22:59:05] diederik: https://gerrit.wikimedia.org/r/p/analytics/udp-filters.git [22:59:54] hasharCall: is it working? [22:59:56] maybe text submitted using the CLI is not formatted by gerrit [22:59:58] seems monospaced to me [22:59:59] no :( [23:00:01] heh [23:00:20] ryan; ty [23:00:25] diederik: yw [23:01:18] I think I will have to wrap it in a 
 :-D
[23:06:32] 	 https://gerrit.wikimedia.org/r/#change,4145   <--- using 
[23:11:02] 	 heading bed now ;)  have fun everyone
[23:12:12] 	 petr/Reedy/whoever:  I'd love it if someone could help me get an actual working account on beta.
[23:12:38] 	 hm?
[23:13:09] 	 Um... I want to to test some api code on beta, but can't edit pages on account of I can't get a login.
[23:13:21] 	 um... s/edit/create/
[23:13:57] 	 Probably because I am doing something dumb
[23:14:31] 	 "Labs uses cookies to log in users. You have cookies disabled. Please enable them and try again"
[23:14:44] 	 Um, well, goddamn, the account that failed a couple of hours ago now works.  so, nevermind!
[23:14:59] 	 3rd time lucky
[23:15:01] 	 lol
[23:15:50] 	 andrewbogott: do you need some userrights?
[23:15:53] 	 Ah, I think the issue is that I can't log into en.
[23:16:05] 	 Took me 3 attempts
[23:16:18] 	 But can log into http://labs.wikimedia.beta.wmflabs.org which should suit me fine.
[23:21:49] 	 PROBLEM Free ram is now: WARNING on deployment-web2 i-00000125 output: Warning: 15% free memory
[23:24:54] 	 Apparently labs can't send emails
[23:26:06] 	 andrewbogott: you're a dev right? I can give you global dev
[23:26:39] 	 I don't know what that means, but I am sort of a dev :)
[23:26:49] 	 RECOVERY Free ram is now: OK on deployment-web2 i-00000125 output: OK: 38% free memory
[23:27:00] 	 I am an 'operations developer' or something like that.
[23:27:13] 	 I think I am unstuck for the moment, in any case.
[23:27:22] 	 heh
[23:27:40] 	 it's "Andrewbogott" right?
[23:27:47] 	 yep
[23:28:38] 	 done
[23:28:52] 	 http://labs.wikimedia.beta.wmflabs.org/wiki/Special:GlobalGroupPermissions/developer - you can do more or less anything you could possibly need to
[23:29:11] 	 nice, thanks.
[23:32:56] * Reedy  is seeing how many sites wil let him login
[23:35:53] 	 sure np
[23:37:08] 	 Reedy: do you have access to deployment-prep on an ssh level?
[23:37:52] 	 Possibly
[23:37:53] 	 I don't know
[23:37:53] 	 ah never mind, I'll just make the settings change on wiki
[23:38:15] 	 I can't edit user-rights on labs which is silly so I wanted it to be made the central wiki instead of meta
[23:38:25] 	 but I could just give myself  Edit user rights of users on other wikis (userrights-interwiki) instead
[23:45:49] 	 lans is sloooooow