[01:08:23] PROBLEM Total processes is now: WARNING on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: PROCS WARNING: 185 processes [01:13:23] RECOVERY Total processes is now: OK on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: PROCS OK: 99 processes [02:38:22] RECOVERY Free ram is now: OK on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: OK: 21% free memory [02:39:52] RECOVERY Free ram is now: OK on swift-be4.pmtpa.wmflabs 10.4.0.127 output: OK: 23% free memory [02:56:20] Would someone be able to add me to the bastion project so I can ssh in? My shell account application was process but I don't appear to be a member and can't ssh in. [02:56:23] PROBLEM Free ram is now: WARNING on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: Warning: 15% free memory [04:04:33] PROBLEM Current Users is now: CRITICAL on cephticon1.pmtpa.wmflabs 10.4.1.56 output: Connection refused by host [04:05:12] PROBLEM Disk Space is now: CRITICAL on cephticon1.pmtpa.wmflabs 10.4.1.56 output: Connection refused by host [04:05:53] PROBLEM Free ram is now: CRITICAL on cephticon1.pmtpa.wmflabs 10.4.1.56 output: Connection refused by host [04:05:53] PROBLEM Current Load is now: CRITICAL on cephticon1.pmtpa.wmflabs 10.4.1.56 output: Connection refused by host [04:07:22] PROBLEM Total processes is now: CRITICAL on cephticon1.pmtpa.wmflabs 10.4.1.56 output: Connection refused by host [04:08:12] PROBLEM dpkg-check is now: CRITICAL on cephticon1.pmtpa.wmflabs 10.4.1.56 output: Connection refused by host [04:12:54] PROBLEM Free ram is now: WARNING on swift-be4.pmtpa.wmflabs 10.4.0.127 output: Warning: 19% free memory [04:14:34] RECOVERY Current Users is now: OK on cephticon1.pmtpa.wmflabs 10.4.1.56 output: USERS OK - 0 users currently logged in [04:15:13] RECOVERY Disk Space is now: OK on cephticon1.pmtpa.wmflabs 10.4.1.56 output: DISK OK [04:15:53] RECOVERY Free ram is now: OK on cephticon1.pmtpa.wmflabs 10.4.1.56 output: OK: 1627% free memory [04:15:53] RECOVERY Current Load is now: OK on cephticon1.pmtpa.wmflabs 10.4.1.56 output: OK - load average: 0.14, 0.66, 0.59 [04:17:23] RECOVERY Total processes is now: OK on cephticon1.pmtpa.wmflabs 10.4.1.56 output: PROCS OK: 90 processes [04:18:13] RECOVERY dpkg-check is now: OK on cephticon1.pmtpa.wmflabs 10.4.1.56 output: All packages OK [04:54:23] PROBLEM Free ram is now: CRITICAL on aggregator1.pmtpa.wmflabs 10.4.0.79 output: Critical: 5% free memory [04:59:23] PROBLEM Free ram is now: WARNING on aggregator1.pmtpa.wmflabs 10.4.0.79 output: Warning: 7% free memory [06:28:53] PROBLEM Total processes is now: WARNING on parsoid-roundtrip4-8core.pmtpa.wmflabs 10.4.0.39 output: PROCS WARNING: 151 processes [06:42:53] PROBLEM dpkg-check is now: CRITICAL on mobile-solr3.pmtpa.wmflabs 10.4.1.35 output: DPKG CRITICAL dpkg reports broken packages [06:47:52] RECOVERY dpkg-check is now: OK on mobile-solr3.pmtpa.wmflabs 10.4.1.35 output: All packages OK [06:53:52] RECOVERY Total processes is now: OK on parsoid-roundtrip4-8core.pmtpa.wmflabs 10.4.0.39 output: PROCS OK: 146 processes [10:26:53] 12/24/2012 - 10:26:52 - Creating a home directory for tb at /export/keys/tb [10:31:43] 12/24/2012 - 10:31:43 - Updating keys for tb at /export/keys/tb [10:51:28] 12/24/2012 - 10:51:28 - Updating keys for tb at /export/keys/tb [11:11:19] 12/24/2012 - 11:11:18 - Updating keys for tb at /export/keys/tb [11:44:48] * Damianz wonders what Santa_Claus is bringing him [11:45:18] Remind me, have you been a good or a bad guy? [11:46:39] Always bad :D [11:51:25] 12/24/2012 - 11:51:25 - Updating keys for tb at /export/keys/tb [13:21:35] 12/24/2012 - 13:21:34 - Updating keys for tb at /export/keys/tb [15:15:24] can't login, can't reset password: "There was either an authentication database error or you are not allowed to update your external account." [16:40:47] any admins around? still can't login & pswd reset is broken [18:01:03] Yurik, still there? [18:01:12] yep [18:01:24] OK, let me see what I can find in a log file. [18:01:41] fyi, the last time we encountered this problem it was because usernames are case-sensitive. [19:20:01] Ryan_Lane, ping me when you're back? Trying to sort out Yurik's password-changing issues. I suspect it's something to do with case, he's 'yurik' in ldap but the logfile keeps calling him 'Yurik' (even though he says he is typing 'yurik' in the form, and I believe him) [19:30:14] 12/24/2012 - 19:30:13 - Creating a project directory for mediawiki-api [20:10:53] PROBLEM Current Load is now: CRITICAL on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: Connection refused by host [20:11:34] PROBLEM Current Users is now: CRITICAL on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: Connection refused by host [20:12:13] PROBLEM Disk Space is now: CRITICAL on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: Connection refused by host [20:12:53] PROBLEM dpkg-check is now: CRITICAL on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: Connection refused by host [20:13:03] PROBLEM Free ram is now: CRITICAL on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: Connection refused by host [20:14:23] PROBLEM Total processes is now: CRITICAL on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: Connection refused by host [20:21:32] RECOVERY Current Users is now: OK on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: USERS OK - 0 users currently logged in [20:22:12] RECOVERY Disk Space is now: OK on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: DISK OK [20:22:52] RECOVERY dpkg-check is now: OK on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: All packages OK [20:23:02] RECOVERY Free ram is now: OK on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: OK: 896% free memory [20:24:22] RECOVERY Total processes is now: OK on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: PROCS OK: 84 processes [20:25:52] RECOVERY Current Load is now: OK on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: OK - load average: 0.02, 0.52, 0.54 [20:34:22] RECOVERY Free ram is now: OK on aggregator1.pmtpa.wmflabs 10.4.0.79 output: OK: 919% free memory [20:55:53] PROBLEM dpkg-check is now: CRITICAL on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: DPKG CRITICAL dpkg reports broken packages [20:57:39] andrewbogott: password changing issue? [20:57:48] I thought he changed it successfully the other day [20:58:04] I thought so too. [20:58:07] Yurik, still here? [20:58:28] mediawiki always changes the first character to uppercase [20:58:39] Ah, ok, so that part is a red herring... [20:58:52] his cn is: Yuri Astrakhan [20:58:55] not Yurik [21:00:05] yep [21:00:22] is that what you entered as a real name? [21:00:28] in the account creation form? [21:00:38] i don't think i had that form [21:00:41] ah [21:00:46] someone else created your account? [21:00:48] i already had an account [21:00:51] I see [21:00:58] i had an SVN acct [21:01:07] what do you actually want as your wiki username? [21:01:11] Yurik? [21:01:34] what's the common standard? [21:01:40] whatever you want it to be [21:01:50] it's weird. I linked your account [21:01:56] did you change your real name in the wiki? [21:02:00] in preferences? [21:02:03] I bet it wrote to cn [21:02:05] yes [21:02:17] i changed the prefs to show full name [21:02:18] that's the issue. I need to disable that feature [21:02:22] but login shuold stay yurik [21:02:26] lol [21:02:45] can i have login as yurik and full name as that? [21:02:53] or do i need to have just one? [21:03:17] Ryan_Lane: So, in ldap… 'cn' is the wiki login name, and 'uid' is the shell name? [21:03:23] yep [21:03:30] if ( is_string( $this->realname ) ) { $values["cn"] = $this->realname; } [21:03:33] And sn… neither of the above? [21:03:37] ^^ that's why that's broken [21:03:50] sn is a required attribute in the inetorgperson objectclass [21:03:56] which is the only reason it's even filled in [21:04:08] we don't use it for anything [21:04:16] it's such an absurd requirement of that objectclass [21:04:37] so, I commented out that line in LdapAuthentication [21:05:09] ryan, i see you have your user account as full name - is that how you login? [21:05:41] yep [21:05:52] I use my full name in wiki and my username via shell [21:06:05] in any case, i think i would rather stick with "yurik", and if anyone is interested, my name is on my page [21:06:38] hi Ryan_Lane [21:06:47] https://gerrit.wikimedia.org/r/#/settings/ shows "yurik" for username and "Yurik" for Full Name [21:06:58] 12/24/2012 - 21:06:58 - Updating keys for yurik at /export/keys/yurik [21:07:09] is Full Name really the account name? [21:07:12] Nikerabbit: howdy [21:07:21] Yurik: I set yours back to Yurik [21:07:26] Ryan_Lane: I was going to say howdy! [21:07:33] which one is used for "User:xxx" ? [21:07:35] and now if you update realname, it won't break [21:07:39] the full name [21:07:49] Yurik: yours is Yurik [21:07:56] ' been enjoying christmas eve [21:08:02] I specifically picked Ryan Lane as mine [21:08:13] Nikerabbit: it's been good. how's yours? [21:08:56] Yurik: Let's figure out about your ssh keys while we're at it... [21:09:06] Can you log out and in and see if you still can't upload? [21:09:12] Ryan_Lane: interesting weather, physical exercise and opening of presents... day's finished in few hours [21:09:12] sec, i might still be getting reset error [21:09:28] Yurik: you're using "Yurik" now, right? :) [21:09:41] Nikerabbit: heh. I still need to buy presents :) [21:09:47] nope, all is good [21:09:57] "yurik"! [21:10:06] same same for the wiki [21:10:08] that's ok, right? [21:10:10] yes [21:10:12] grrr [21:10:13] first letter is always capitalized [21:10:21] for the shell it's case sensitive [21:11:34] Ryan_Lane: key imported ok too. guess they were linked [21:11:42] yeah [21:11:46] it couldn't find your user account [21:11:50] 12/24/2012 - 21:11:50 - Updating keys for yurik at /export/keys/yurik [21:11:58] * Ryan_Lane goes to enter a bug [21:12:17] yei, thanks guys! [21:13:49] yw. sorry about the problems [21:15:17] Now that I look back through the ldap log I can see all the pieces that should've diagnosed this… I need more ldap practice. [21:15:32] heh [21:16:12] I started to re-read an ldap overview on Friday but immediately fell asleep for 90 minutes [21:16:19] hahaha [21:16:25] because it was boring? [21:16:28] ldap *is* boring [21:16:45] 12/24/2012 - 21:16:44 - Updating keys for yurik at /export/keys/yurik [21:17:06] Because it was boring or because I have a special gland in my body tasked with ensuring that I never understand ldap [21:17:11] :D [21:17:27] the basics of ldap are easy. think of it like an object relational database [21:17:34] it stores objects in a tree [21:17:43] those objects have classes and attributes [21:17:57] the classes are the specified schema for the objects [21:18:44] Are there any decent visualization tools? I understand about it being tree shaped but can never put that into practice when looking at actual data. [21:18:56] hm. not really [21:19:09] there's what's called a DIT (directory information tree) [21:19:20] do i have ssh access now? I tried yurik@bastion.wmflabs.org [21:19:23] it starts with a base, which in our case is: dc=wikimedia,dc=org [21:19:32] Yurik: let me make sure you have shell rights [21:19:41] He does. Or did, an hour ago. [21:19:46] ah ok [21:19:50] i get pubkey [21:19:57] perm denied [21:19:58] let me check logs [21:20:14] Yurik: You're just logging into bastion? [21:20:19] yep [21:20:21] Yurik: you need to use yurik, not Yurik for shell [21:20:27] i tried both [21:20:43] Yurik: try the lowercase one for me [21:20:45] I'm tailing the logs [21:21:18] let me try it through putty [21:21:19] Ryan_Lane: The base is arbitrary, right? And a given db can contain multiple trees with different bases? [21:21:40] 12/24/2012 - 21:21:39 - Updating keys for yurik at /export/keys/yurik [21:21:45] Yurik: ah, your key hadn't been added yet [21:21:51] still isn't there [21:21:54] a script adds it [21:22:06] i just added both keys i use [21:22:10] about 5 minago [21:22:36] Yurik: yeah, the script looks broken [21:22:41] next time i don't start loging in without a tamborine [21:23:01] we recently changed the keys stuff [21:23:15] i heard in the ru-net that it helps [21:23:17] andrewbogott: script has a traceback :D [21:23:35] in rename user? [21:23:36] weird [21:24:01] ah [21:24:13] Related to recent changes? [21:24:15] seems the homedirectory stuff is running all of the functions [21:25:09] it's trying to rename a user [21:26:47] I modifed /usr/local/lib/user-management/homedirectorymanager.py [21:27:09] to disable everything except for key updating and directory creation [21:27:12] It was a collision? Renaming a user to an existing username? [21:27:24] I think so, yes [21:27:44] Yurik: ok, it should work now [21:27:59] yei [21:28:21] Isn't that module used by other scripts? [21:28:47] nope [21:29:03] it was originally used to manage the home directories and the keys [21:29:07] now just the keys [21:29:09] andrewbogott: am i the sys admin / net admin for api? [21:29:15] Yurik: Yes. [21:30:50] that script needs better error checking [21:31:01] it should really skip something if it fails out, and log it [21:31:21] Who creates homedirs now? Is that in manage-volumes? [21:31:31] pam_mkhomedir [21:31:53] when a user logs into an instance, if the home directory doesn't exist, the instance creates it [21:32:02] Ah, ok. That's better. [21:32:05] yeah [21:32:53] Maybe I will gitify/puppetize these scripts after lunch so we can track our hacks [21:32:58] andrewbogott: i can add sudo policy and puppet groups, but i cant create instances, manage addr or security. Do I need to add a sudo group first? [21:33:11] andrewbogott: cool. they are already in puppet [21:33:13] lemme look. You should be able to create instances... [21:33:13] they pull from svn [21:33:35] Ryan_Lane: But our custom changes don't get clobbered... [21:33:39] How does that work? [21:34:03] because it checks out a specific revision [21:34:09] and uncommitted changes stay in svn [21:34:58] Yurik: Are you here? https://labsconsole.wikimedia.org/w/index.php?title=Special:NovaInstance&action=create&project=mediawiki-api®ion=pmtpa [21:35:34] i was in "manage instances" on the left menu, and there weren't any links [21:35:43] Hm. [21:35:52] This is dumb, but -- try logging out and in afresh [21:36:07] there *is* a bug there. I wonder if I added it to bugzilla [21:36:32] yep, now i see them [21:36:34] facepalm [21:36:43] keystone is a pain in the ass [21:37:07] Yurik: Right now we're chock full of these inconsistent state problems… so a fresh login is always worth trying. [21:37:17] :) [21:37:35] Yurik: I'm going to go to lunch, as long as you're unstuck [21:37:38] its ok, i just feel like i should do shaman dancing to make it work better [21:37:43] one sec [21:38:16] yeah, an instance got created [21:38:18] you're a lucky one, Yurik [21:38:20] don't worry [21:38:27] you're hitting like every bug possible [21:38:31] yep [21:38:34] sounds that way :D [21:38:38] this has been 3rd so far [21:38:51] this is actually a pretty rare day :) [21:39:03] oh, sorry, 4th [21:39:10] Yurik: It'll take a few minutes for the instance to come up; you can watch progress in the log. [21:39:20] Yurik: you hit a new one now? [21:39:37] yeah. the instance isn't actually fully up till puppet finishes running [21:39:41] no, just forgot about one i had before, so fixed counting :) [21:39:46] But, also, probably you'll want to create yourself a 'web' security group and then start a new instance with that… [21:39:56] Yurik: Can't add security groups to existing instances, alas. [21:40:02] saltstack just added a feature to act on events, so now I'll be able to report in IRC when an instance is fully finished [21:40:05] thx, killing this one [21:40:13] if mediawiki ever gets a proper notification framework, I can also make it report there, too [21:40:21] how big for an almost nodata wiki do i need? [21:40:30] small [21:40:34] m1.small [21:40:34] Yurik: size 'small' always works for me. [21:40:42] thankfully I removed tiny [21:40:52] i just saw very small [21:40:53] OK, back in 30 or so. [21:40:59] thx! [21:41:47] when i delete an instance, does it delete right away? [21:41:59] no. it takes a little bit [21:42:12] it should be gone within a few minutes [21:44:53] PROBLEM host: api.pmtpa.wmflabs is DOWN address: 10.4.1.58 CRITICAL - Host Unreachable (10.4.1.58) [22:00:52] PROBLEM Current Load is now: CRITICAL on api1.pmtpa.wmflabs 10.4.1.59 output: Connection refused by host [22:00:52] PROBLEM Free ram is now: CRITICAL on api1.pmtpa.wmflabs 10.4.1.59 output: Connection refused by host [22:01:32] PROBLEM Current Users is now: CRITICAL on api1.pmtpa.wmflabs 10.4.1.59 output: Connection refused by host [22:02:12] PROBLEM Disk Space is now: CRITICAL on api1.pmtpa.wmflabs 10.4.1.59 output: Connection refused by host [22:02:22] PROBLEM Total processes is now: CRITICAL on api1.pmtpa.wmflabs 10.4.1.59 output: Connection refused by host [22:04:32] PROBLEM dpkg-check is now: CRITICAL on api1.pmtpa.wmflabs 10.4.1.59 output: Connection refused by host [22:09:32] RECOVERY dpkg-check is now: OK on api1.pmtpa.wmflabs 10.4.1.59 output: All packages OK [22:10:52] RECOVERY Current Load is now: OK on api1.pmtpa.wmflabs 10.4.1.59 output: OK - load average: 0.19, 0.87, 0.69 [22:10:52] RECOVERY Free ram is now: OK on api1.pmtpa.wmflabs 10.4.1.59 output: OK: 1043% free memory [22:11:32] RECOVERY Current Users is now: OK on api1.pmtpa.wmflabs 10.4.1.59 output: USERS OK - 0 users currently logged in [22:12:12] RECOVERY Disk Space is now: OK on api1.pmtpa.wmflabs 10.4.1.59 output: DISK OK [22:12:22] RECOVERY Total processes is now: OK on api1.pmtpa.wmflabs 10.4.1.59 output: PROCS OK: 83 processes [22:25:53] how do i connect to the instance? I just created an instance, but ssh-ing to it gives me publckey permission denied [22:26:33] and the only docs i found said something about Admin account [22:27:19] Yurik -- you need to forward your key from your local machine to bastion. [22:27:31] Are you on windows or mac or linux? [22:27:35] win [22:27:43] is that part of ssh in putty? [22:28:16] Um… I don't know how to do it from windows. I feel like folks are using a second tool besides putty. [22:28:20] Lemme look... [22:28:31] yeah, there is something called agent forwarding [22:28:33] checking [22:29:17] yep, worked , thanks! [22:29:47] Cool. [22:34:46] i connected to the instance with ssh -A ip, but when i do sudo, it still asks for the password [22:35:00] Yep, that's correct behavior. [22:35:08] Use the same password that you use to log in to labsconsole [22:35:14] gotcha [22:35:33] nope, i'm not allowed to run sudo there [22:35:54] sudo puppetd -tv [22:42:13] andrewbogott: ^^ [22:42:33] Yurik: which instance? [22:42:37] Instance name? [22:42:38] api1 [22:42:40] is this a new project? [22:42:45] yep [22:43:06] andrew created it today [22:43:11] should probably make the sudo policy ALL, ALL, ALL [22:43:15] and don't specify instance name [22:43:34] until we fix all of the instance naming, specifying instance doesn't work [22:43:34] Sysadmins don't have sudo by default? [22:43:37] no [22:43:44] i should create a sudo policy [22:43:47] ok [22:43:49] oops, sorry to mislead. [22:43:52] yes, specify all instances [22:44:27] any commands or options? [22:44:30] ALL [22:44:57] I think I may have a bug open for making a default sudo policy [22:45:05] since more often than not people want sudo for everyone [22:45:49] worked, thx [22:46:07] could i get an ip alloc pls [22:47:28] sure. what's the project name? [22:47:35] mediawiki-api [22:47:49] i wanted "api" but andrew just wouldn't have it that way [22:47:56] :) [22:49:46] heh [22:49:49] probably a good idea [22:50:10] ok, you can allocate now one [22:53:59] Ryan_Lane: Is there a reason to put these tools in operations/debs/wikimedia-ldap-tools vs just sticking them in puppet/files? [22:54:10] I don't plan on making a deb out of them, and operations/debs/wikimedia-ldap-tools is currently 100% empty [22:54:34] thx [23:05:15] bleh - installed role::mediawiki-install::labs and now if i go to http://api.beta.wmflabs.org/ it works, but going to http://api.beta.wmflabs.org/w autoredirects to http://www.api.com/wiki/Main_Page [23:06:19] Did you already set labs_mediawiki_hostname? [23:06:54] In theory that should fix things, although I'm not sure if it works to change that setting after the wiki is installed… you might have to clean some things up. [23:06:56] i set it to "api" and later changed to "api1" without any visible effect [23:07:05] what should it be set to? [23:07:54] "fully qualified hostname of your instance" [23:08:05] So if you want the site at api.beta.wmflabs.org [23:08:51] And, try wiping out your /srv/ dir so that puppet is forced to reinstall and reconfigure the wiki. [23:08:53] rerunning sudo puppetdtv [23:09:02] sigh [23:09:18] I don't think puppet will re-run the wiki autoconfig; that seems dangerous. [23:11:22] do i have root on the instance? [23:11:27] it gives me perm denied [23:11:48] sorry for all the silly q [23:12:32] No problem... [23:12:41] What do you mean by 'have root'? You have sudo sorted out right? [23:12:45] yes [23:12:57] but i can't just do rm -f -r for /srv [23:13:07] no, but… sudo rm -rf [23:13:07] i meant /srv/mediawiki [23:13:31] oh, silly me, i used su for some reason [23:14:23] should i get just the mediawiki package, or other things like cache, etc [23:14:54] Just naked wikimedia, plus a few security extensions [23:15:00] andrewbogott: puppet is fine [23:15:13] it was moved to svn because at the time no one other than ops could modify them [23:15:19] what security ext? [23:16:12] nuke, spamblocker, confirmedit [23:16:20] *spamblacklist [23:16:35] but they are not part of the instance config, right? [23:16:47] nope [23:16:48] or you mean get them through git [23:16:54] They're in a subdir anyway. [23:16:59] Yeah, they're fetched via git [23:17:04] got it, thx [23:21:42] http://api.beta.wmflabs.org/wiki/Main_Page -- results... [23:21:50] all sub-paths messed up [23:21:52] :( [23:22:14] oh [23:22:15] yeah [23:22:28] andrewbogott: wanted to mention, there's a config error [23:22:44] i feel like a guinepig more and more [23:22:53] heh [23:23:04] Yurik: in /srv/mediawiki/orig/LocalSettings.php [23:23:08] you want to set: $wgScriptPath = "/w"; [23:23:30] I went through this yesterday [23:23:43] Yurik: in some ways you are [23:23:56] it's been a while since we've linked an svn account [23:25:42] I recently fixed the preferences form to allow updates to ldap again, so you're the first to change real name ;) [23:27:00] it's nice to have someone walk all the way through things again. we're obviously catching some bugs that others may not be reporting [23:27:04] didn't help for some reasno [23:27:17] script = /w [23:27:19] worked for me [23:27:25] force-refresh in your browser [23:28:28] oh yeah, cool. Can i setup remote php debugging? also a memcache etc? [23:28:49] i saw a package in the config list - memcache [23:28:50] memcache is already set up and configured [23:28:54] a, cool [23:28:58] so is apc [23:29:10] apc? [23:29:14] php-apc [23:29:24] it's a php opcode cache [23:29:32] what about remote debugging? [23:29:36] not configured [23:29:43] you'll need to install the package [23:29:43] has anyone ever done that? [23:29:50] I'm sure someone has :) [23:29:56] but its doable. cool [23:30:12] it has a public IP, so it should definitely be doable, yeah [23:30:24] time to host some nutty pixs [23:31:03] i haven't setup php remote dbg yet, might be a fun experience. [23:31:19] hope my eclipse doesn't barf [23:35:53] PROBLEM Current Load is now: CRITICAL on testlabs-abogott-dev.pmtpa.wmflabs 10.4.1.20 output: Connection refused by host [23:36:33] PROBLEM Current Users is now: CRITICAL on testlabs-abogott-dev.pmtpa.wmflabs 10.4.1.20 output: Connection refused by host [23:37:13] PROBLEM Disk Space is now: CRITICAL on testlabs-abogott-dev.pmtpa.wmflabs 10.4.1.20 output: Connection refused by host [23:37:53] PROBLEM dpkg-check is now: CRITICAL on testlabs-abogott-dev.pmtpa.wmflabs 10.4.1.20 output: Connection refused by host [23:38:03] PROBLEM Free ram is now: CRITICAL on testlabs-abogott-dev.pmtpa.wmflabs 10.4.1.20 output: Connection refused by host [23:39:23] PROBLEM Total processes is now: CRITICAL on testlabs-abogott-dev.pmtpa.wmflabs 10.4.1.20 output: Connection refused by host [23:46:33] RECOVERY Current Users is now: OK on testlabs-abogott-dev.pmtpa.wmflabs 10.4.1.20 output: USERS OK - 2 users currently logged in [23:47:12] RECOVERY Disk Space is now: OK on testlabs-abogott-dev.pmtpa.wmflabs 10.4.1.20 output: DISK OK [23:47:52] RECOVERY dpkg-check is now: OK on testlabs-abogott-dev.pmtpa.wmflabs 10.4.1.20 output: All packages OK [23:48:02] RECOVERY Free ram is now: OK on testlabs-abogott-dev.pmtpa.wmflabs 10.4.1.20 output: OK: 568% free memory [23:48:48] do i need to use gerrit (git review) if at first i would rather have a server-stored branch that the instance will pull from. And once code is stable, it will get pushed with `git review` [23:49:12]