[01:16:11] PROBLEM Current Load is now: WARNING on deployment-nfs-memc i-000000d7 output: WARNING - load average: 5.09, 7.15, 5.16 [01:26:11] RECOVERY Current Load is now: OK on deployment-nfs-memc i-000000d7 output: OK - load average: 0.75, 2.73, 4.02 [01:29:39] <^demon|away> Ryan_Lane: I think they're planning to release 2.3 final tomorrow. [01:29:47] cool [01:31:09] <^demon|away> I'll upgrade gerrit-dev from rc0 to the final release when they do. [01:31:26] <^demon|away> With any bit of luck we'll not hit any regressions and can upgrade the production site soon. [01:33:11] the change to drafts won't affect us any? [01:34:34] <^demon|away> Nobody's been using drafts yet. [01:34:41] <^demon|away> So shouldn't hurt anything [01:34:50] * Ryan_Lane nods [02:15:11] mod_rewrite makes me angry [02:42:10] RECOVERY Current Load is now: OK on hugglewa-1 i-000001e0 output: OK - load average: 0.33, 0.52, 0.24 [02:42:50] RECOVERY Current Users is now: OK on hugglewa-1 i-000001e0 output: USERS OK - 0 users currently logged in [02:42:50] RECOVERY Disk Space is now: OK on hugglewa-1 i-000001e0 output: DISK OK [02:42:50] RECOVERY Free ram is now: OK on hugglewa-1 i-000001e0 output: OK: 88% free memory [02:43:44] RECOVERY Total Processes is now: OK on hugglewa-1 i-000001e0 output: PROCS OK: 81 processes [02:44:20] RECOVERY dpkg-check is now: OK on hugglewa-1 i-000001e0 output: All packages OK [03:38:10] PROBLEM Puppet freshness is now: CRITICAL on puppet-lucid i-00000080 output: Puppet has not run in last 20 hours [03:50:50] PROBLEM Free ram is now: WARNING on nova-daas-1 i-000000e7 output: Warning: 14% free memory [03:51:00] PROBLEM Free ram is now: WARNING on test-oneiric i-00000187 output: Warning: 16% free memory [03:56:00] PROBLEM Free ram is now: WARNING on utils-abogott i-00000131 output: Warning: 15% free memory [03:56:00] PROBLEM Free ram is now: WARNING on orgcharts-dev i-0000018f output: Warning: 14% free memory [04:10:50] PROBLEM Free ram is now: CRITICAL on test3 i-00000093 output: Critical: 3% free memory [04:10:50] PROBLEM Free ram is now: CRITICAL on nova-daas-1 i-000000e7 output: Critical: 5% free memory [04:11:00] PROBLEM Free ram is now: CRITICAL on orgcharts-dev i-0000018f output: Critical: 5% free memory [04:11:00] PROBLEM Free ram is now: CRITICAL on test-oneiric i-00000187 output: Critical: 4% free memory [04:15:50] RECOVERY Free ram is now: OK on test3 i-00000093 output: OK: 96% free memory [04:16:00] PROBLEM Free ram is now: CRITICAL on utils-abogott i-00000131 output: Critical: 3% free memory [04:16:00] RECOVERY Free ram is now: OK on test-oneiric i-00000187 output: OK: 96% free memory [04:20:50] RECOVERY Free ram is now: OK on nova-daas-1 i-000000e7 output: OK: 93% free memory [04:21:00] RECOVERY Free ram is now: OK on utils-abogott i-00000131 output: OK: 97% free memory [04:21:00] RECOVERY Free ram is now: OK on orgcharts-dev i-0000018f output: OK: 95% free memory [07:10:34] what is name of Chris on labs [07:10:40] Cmmcmahon [07:10:43] doesn't exist [08:50:00] RECOVERY Disk Space is now: OK on deployment-transcoding i-00000105 output: DISK OK [08:58:00] PROBLEM Disk Space is now: WARNING on deployment-transcoding i-00000105 output: DISK WARNING - free space: / 73 MB (5% inode=53%): [11:40:58] New patchset: Dzahn; "wikstats - make https optional again" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/4259 [11:41:10] New review: gerrit2; "Lint check passed." [operations/puppet] (test); V: 1 - https://gerrit.wikimedia.org/r/4259 [13:04:57] New patchset: Dzahn; "initial import of index,display and rank php files from wikistats" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/4264 [13:05:09] New review: gerrit2; "Lint check passed." [operations/puppet] (test); V: 1 - https://gerrit.wikimedia.org/r/4264 [13:39:10] PROBLEM Puppet freshness is now: CRITICAL on puppet-lucid i-00000080 output: Puppet has not run in last 20 hours [14:41:47] New patchset: Dzahn; "remove cookie handling stuff from display.php" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/4273 [14:41:59] New review: gerrit2; "Lint check passed." [operations/puppet] (test); V: 1 - https://gerrit.wikimedia.org/r/4273 [14:45:18] New review: Hashar; "shouldn't that be in its own repository then have puppet to git pull from the repo?" [operations/puppet] (test); V: 0 C: 0; - https://gerrit.wikimedia.org/r/4264 [14:45:24] away [15:39:13] New patchset: Dzahn; "wikistats - import functions.php" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/4277 [15:39:25] New patchset: Dzahn; "wikistats - import grandtotal.php" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/4278 [15:39:41] New patchset: Dzahn; "wikistats - add http_status_codes array" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/4279 [15:39:54] New patchset: Dzahn; "wikistats - import footer and sortswitch" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/4280 [15:40:07] New patchset: Dzahn; "wikistats - import api.php, csv dumper" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/4281 [15:40:19] New review: gerrit2; "Lint check passed." [operations/puppet] (test); V: 1 - https://gerrit.wikimedia.org/r/4277 [15:40:19] New review: gerrit2; "Lint check passed." [operations/puppet] (test); V: 1 - https://gerrit.wikimedia.org/r/4278 [15:40:19] New review: gerrit2; "Lint check passed." [operations/puppet] (test); V: 1 - https://gerrit.wikimedia.org/r/4279 [15:40:19] New review: gerrit2; "Lint check passed." [operations/puppet] (test); V: 1 - https://gerrit.wikimedia.org/r/4280 [15:40:20] New review: gerrit2; "Lint check passed." [operations/puppet] (test); V: 1 - https://gerrit.wikimedia.org/r/4281 [15:52:39] New review: Dzahn; "I'd be fine with that as well. Just wanted to start fixing these imported files and have them in her..." [operations/puppet] (test); V: 0 C: 0; - https://gerrit.wikimedia.org/r/4264 [16:52:10] PROBLEM Free ram is now: CRITICAL on bots-3 i-000000e5 output: Critical: 5% free memory [17:07:10] PROBLEM Free ram is now: WARNING on bots-3 i-000000e5 output: Warning: 8% free memory [17:12:10] RECOVERY Free ram is now: OK on bots-3 i-000000e5 output: OK: 39% free memory [17:35:45] Hi [17:35:52] How do I get a new project created? [17:36:02] <^demon> Gerrit project or labs project? [17:36:42] Labs one [17:46:45] !help [17:46:45] !documentation for labs !wm-bot for bot [17:47:13] !wm-bot [17:47:14] http://meta.wikimedia.org/wiki/WM-Bot [17:57:32] andrewbogott: Remind me, do you create labs projects for people? What's the process for requesting those? [17:58:14] I can, yes. [17:58:24] And the process is pretty lax. What do you need? [17:58:38] Oh, wait, reading the scrollback... [17:59:27] vvv, do you have a labs account already? And, what will you do with your project? [17:59:54] andrewbogott: I do; I want a project to set up a test installation of OpenGrok [18:00:49] does that relate to some sort of mediawiki/wikimedia undertaking? [18:01:11] It's an alternative repo viewer that he wants to use on gerrit.wm.o [18:01:37] Ah, ok. [18:01:45] Just a second... [18:03:51] vvv, what's your labs username? [18:04:06] SSH is vvv [18:04:12] Wiki is Victor Vasiliev [18:06:18] vvv: Do you see the project on your 'manage instances' page now? [18:06:33] Yes [18:07:18] great. Lemme know if you need help with anything else. [18:07:40] Ryan_Lane, did you get anyone on openstack-dev to comment about whether we need to switch to keystone for essex? [18:08:33] oh [18:08:39] so, they left it in for migration purposes [18:09:01] That's good, right? [18:09:10] Or did they leave it in, but broken? [18:09:13] RoanKattouw, vvv: does it integrate properly with gerrit? [18:09:23] Ryan_Lane: it should, according to docs [18:09:30] ah, good [18:09:37] vvv: make sure to test it with private repos! [18:09:39] Hmm, maybe it could be installed in the existing gerrit-dev VM? [18:09:51] meaning repos that have read restrictions [18:10:00] we have some of those in production [18:10:18] I'm starting to think I want to move the private repo out of gerrit [18:10:35] and onto an NFS share that's shared read-only to all labs instances [18:10:41] it just complicates things to have it in gerrit [18:10:51] Ryan_Lane: ^demon and I have another Gerrit perms issue to discuss with you in a minute [18:10:56] * Ryan_Lane nods [18:11:05] Which should probably happen in private [18:11:36] sure [18:20:34] petan: thanks for the help, I can run a tail on commonswiki_errors.log now, this should be helpful [19:49:41] "There were no Nova credentials found for your user account. Please ask a Nova administrator to create credentials for you" [19:49:53] I often get that error message [19:50:40] Ryan_Lane: any ideas how to fix it and what causes that? [19:51:36] log out and back in [19:51:47] vvv: fix mediawiki's piss-poor authentication code? [19:52:02] something is destroying your session [19:52:16] but only enough to wipe out your LDAP info [19:52:28] Sounds awful [19:53:21] What are m* and s* instance types? [19:57:26] s are storage [19:57:32] m probably stands for machine [19:57:43] it's the default types that come with openstack [19:57:47] I recommend using m types [19:58:03] s types were there for before we had project storage at /data/project [20:01:29] hmm, db error upon attempting to login at http://en.wikipedia.beta.wmflabs.org known issue? [20:02:33] chrismcmahon: Happens to me too, but I don't think anyone has diagnosed or fixed it. [20:03:28] andrewbogott: labs commons seems OK, but en not so much [20:03:34] yep, same for me. [20:03:52] I'm looking to do a cross-wiki media test on those two [20:05:15] This is probably a petan problem, and petan is probably sleeping. [20:13:36] * chrismcmahon goes to Plan B [20:13:43] PROBLEM Current Load is now: CRITICAL on opengrok-web i-000001e1 output: Connection refused by host [20:14:23] PROBLEM Current Users is now: CRITICAL on opengrok-web i-000001e1 output: CHECK_NRPE: Error - Could not complete SSL handshake. [20:17:33] PROBLEM dpkg-check is now: CRITICAL on opengrok-web i-000001e1 output: CHECK_NRPE: Error - Could not complete SSL handshake. [20:22:43] PROBLEM Free ram is now: CRITICAL on opengrok-web i-000001e1 output: CHECK_NRPE: Error - Could not complete SSL handshake. [20:22:53] PROBLEM Total Processes is now: CRITICAL on opengrok-web i-000001e1 output: CHECK_NRPE: Error - Could not complete SSL handshake. [20:23:03] PROBLEM Disk Space is now: CRITICAL on opengrok-web i-000001e1 output: CHECK_NRPE: Error - Could not complete SSL handshake. [20:27:08] I have a shared home directory across multiple instances in the same project, right? [20:29:43] Yes [20:38:43] RECOVERY Current Load is now: OK on opengrok-web i-000001e1 output: OK - load average: 0.97, 0.38, 0.17 [20:39:23] RECOVERY Current Users is now: OK on opengrok-web i-000001e1 output: USERS OK - 1 users currently logged in [20:42:27] Ryan_Lane: what shall I do with a software not in Ubuntu repo? Build it from sources or just download binary JARs? [20:42:33] RECOVERY dpkg-check is now: OK on opengrok-web i-000001e1 output: All packages OK [20:42:43] RECOVERY Free ram is now: OK on opengrok-web i-000001e1 output: OK: 90% free memory [20:42:53] RECOVERY Total Processes is now: OK on opengrok-web i-000001e1 output: PROCS OK: 86 processes [20:43:03] RECOVERY Disk Space is now: OK on opengrok-web i-000001e1 output: DISK OK [20:43:20] vvv: ummm [20:43:31] vvv: depends on what you are trying to do [20:43:43] Well, I want to get the app running [21:03:47] New patchset: Ryan Lane; "Use the rrdns entries for the glusterfs clusters" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/4307 [21:03:59] New review: gerrit2; "Lint check passed." [operations/puppet] (test); V: 1 - https://gerrit.wikimedia.org/r/4307 [21:04:32] New review: Ryan Lane; "(no comment)" [operations/puppet] (test); V: 0 C: 2; - https://gerrit.wikimedia.org/r/4307 [21:04:34] Change merged: Ryan Lane; [operations/puppet] (test) - https://gerrit.wikimedia.org/r/4307 [21:35:04] New patchset: Ryan Lane; "Adding another wiki in for our gsoc student" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/4309 [21:35:16] New review: gerrit2; "Lint check passed." [operations/puppet] (test); V: 1 - https://gerrit.wikimedia.org/r/4309 [21:35:19] New review: Ryan Lane; "(no comment)" [operations/puppet] (test); V: 0 C: 2; - https://gerrit.wikimedia.org/r/4309 [21:35:21] Change merged: Ryan Lane; [operations/puppet] (test) - https://gerrit.wikimedia.org/r/4309 [21:45:59] New patchset: Ryan Lane; "Change the controller name for labs instances" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/4311 [21:46:11] New review: gerrit2; "Lint check passed." [operations/puppet] (test); V: 1 - https://gerrit.wikimedia.org/r/4311 [21:46:34] New review: Ryan Lane; "(no comment)" [operations/puppet] (test); V: 0 C: 2; - https://gerrit.wikimedia.org/r/4311 [21:46:36] Change merged: Ryan Lane; [operations/puppet] (test) - https://gerrit.wikimedia.org/r/4311 [21:51:45] Ryan_Lane: have you used python-mwclient much? [21:51:51] some [21:52:01] adminbot uses it [21:52:35] Is it possible to log /out/, or just log in? [21:52:48] hm [21:52:51] probably just in [21:53:02] Or, my actual question: why does login work for me but only 1 out of ~3 tries? [21:53:25] the 'logout' question is based on the idea that there's some kind of persistent interaction between past and present logins. [21:55:12] Are there any default firewalls or something like that? I have set up a web server and cannot reach it from bastion [21:56:13] Oh, right [21:56:14] Yeah [21:56:25] Go to "Manage security groups" in labsconsole [21:56:49] maybe I should put a warning in on the instance creation form [21:57:03] because no one bothers to read the docs :) [21:57:17] Ryan_Lane: well, I read them [21:57:21] !instances [21:57:22] https://labsconsole.wikimedia.org/wiki/Help:Instances [21:57:35] it mentions on there that you need to create security groups before creating instances [21:57:45] They said something about security groups [21:57:53] But did not explain what exactly they are [21:57:58] !security [21:57:58] https://labsconsole.wikimedia.org/wiki/Help:Security_Groups [21:58:09] it links to that [21:59:24] You can do it after, too [21:59:37] you can't create security groups and add them to instances after [21:59:41] you have to edit default [21:59:46] which is bad practice [22:00:07] Right [22:00:20] I didn't know that, I just edit default [22:00:26] Then again all my projects only have one instance [22:00:30] And that instance runs a web server [22:00:33] * Ryan_Lane nods [22:00:41] it's still bad practice :) [22:00:48] in case you decide later you want another one there [22:00:54] that needs to be private [22:01:07] Yes [23:40:13] PROBLEM Puppet freshness is now: CRITICAL on puppet-lucid i-00000080 output: Puppet has not run in last 20 hours