[00:12:36] ? [00:14:41] anyone help me please [00:15:18] can you not login to the bastion host? [00:15:37] devunt: e.g., ssh -A bastion.wmflabs.org [00:16:07] Permission denied (publickey). [00:16:23] devunt: do you have a key in gerrit? [00:16:29] devunt: an SSH key. [00:16:30] yes. i added [00:16:40] Ryan_Lane: can you help devunt? [00:16:52] I think I'm not in bastion project [00:17:10] maplebed: can you help devunt? [00:32:41] devunt: what will you use labs for? [00:32:56] having access to the bastion doesn't really give you much [00:33:07] you also need access to another project that you'll work in [00:33:35] did you have an already existing project you'd like to work on? or did you want to work on a new project? [00:36:50] Ryan_Lane: Really not now.. [00:38:42] But ok I will think about that [00:39:28] well, it's best for us to not give out shell access unless someone needs it [00:39:51] but when you know what you'd like to work on, we're more than happy to give it to you [00:44:34] per-project sudo! https://labsconsole.wikimedia.org/wiki/Special:NovaSudoer [00:44:37] \o/ [00:57:12] Ryan_Lane: Bastion and Mobile project? [01:06:25] devunt: ah ok. looking to help out with mobile? [01:06:41] I can add you to bastion, but you should talk to a member of the mobile group to be added there [01:08:14] 04/18/2012 - 01:08:13 - Creating a home directory for devunt at /export/home/bastion/devunt [01:09:15] 04/18/2012 - 01:09:14 - Updating keys for devunt [01:10:57] Ryan_Lane: Ok. I'll maunally contact to them. thank you [01:11:19] yw [01:14:23] RECOVERY Current Users is now: OK on localpuppet1 i-0000020b output: USERS OK - 1 users currently logged in [01:15:03] RECOVERY Disk Space is now: OK on localpuppet1 i-0000020b output: DISK OK [01:15:43] RECOVERY Free ram is now: OK on localpuppet1 i-0000020b output: OK: 88% free memory [01:16:43] RECOVERY Current Load is now: OK on localpuppet1 i-0000020b output: OK - load average: 1.56, 0.79, 0.31 [01:16:53] RECOVERY Total Processes is now: OK on localpuppet1 i-0000020b output: PROCS OK: 95 processes [01:17:33] RECOVERY dpkg-check is now: OK on localpuppet1 i-0000020b output: All packages OK [03:36:03] RECOVERY Puppet freshness is now: OK on jenkins2 i-00000102 output: puppet ran at Wed Apr 18 03:35:55 UTC 2012 [03:47:53] PROBLEM Free ram is now: WARNING on orgcharts-dev i-0000018f output: Warning: 13% free memory [03:54:59] PROBLEM Free ram is now: WARNING on nova-daas-1 i-000000e7 output: Warning: 12% free memory [03:55:59] PROBLEM Current Load is now: WARNING on bots-3 i-000000e5 output: WARNING - load average: 10.56, 10.69, 7.21 [03:56:09] PROBLEM dpkg-check is now: CRITICAL on bots-3 i-000000e5 output: CHECK_NRPE: Socket timeout after 10 seconds. [03:57:59] PROBLEM Free ram is now: WARNING on test-oneiric i-00000187 output: Warning: 16% free memory [04:00:59] RECOVERY dpkg-check is now: OK on bots-3 i-000000e5 output: All packages OK [04:02:59] PROBLEM Free ram is now: CRITICAL on orgcharts-dev i-0000018f output: Critical: 5% free memory [04:02:59] PROBLEM Free ram is now: WARNING on utils-abogott i-00000131 output: Warning: 15% free memory [04:07:59] RECOVERY Free ram is now: OK on orgcharts-dev i-0000018f output: OK: 96% free memory [04:12:29] PROBLEM Free ram is now: WARNING on test3 i-00000093 output: Warning: 7% free memory [04:14:59] PROBLEM Free ram is now: CRITICAL on nova-daas-1 i-000000e7 output: Critical: 5% free memory [04:17:29] RECOVERY Free ram is now: OK on test3 i-00000093 output: OK: 96% free memory [04:17:59] PROBLEM Free ram is now: CRITICAL on test-oneiric i-00000187 output: Critical: 3% free memory [04:19:59] RECOVERY Free ram is now: OK on nova-daas-1 i-000000e7 output: OK: 93% free memory [04:20:59] RECOVERY Current Load is now: OK on bots-3 i-000000e5 output: OK - load average: 3.66, 3.77, 4.73 [04:22:59] RECOVERY Free ram is now: OK on test-oneiric i-00000187 output: OK: 97% free memory [04:22:59] PROBLEM Free ram is now: CRITICAL on utils-abogott i-00000131 output: Critical: 3% free memory [04:27:59] RECOVERY Free ram is now: OK on utils-abogott i-00000131 output: OK: 96% free memory [04:53:39] PROBLEM Puppet freshness is now: CRITICAL on nova-production1 i-0000007b output: Puppet has not run in last 20 hours [04:57:39] PROBLEM Puppet freshness is now: CRITICAL on nova-gsoc1 i-000001de output: Puppet has not run in last 20 hours [06:51:05] hi, really liking the puppet stuff wikimedia has going on [06:51:48] so I was wondering about labs.. does it provide an environment that you can take the public wikimedia puppet git repo and apply e.g. parts of it to create standalone testing environments? [06:53:06] I found the example workflow bit in the tech blog post a little weird, there's "configure it by hand" and "[someone?] puppetizes" it separately [06:53:44] but can you actually use labs as a development environment for writing and testing wikimedia-related puppet code? [06:55:23] 'case sysadmin-as-an-open-source-software-development-project is an otherwise fine idea, but the compiler/runtime requirements for "building" the code tend to be kinda harsh :) [06:55:47] (plus some puppet manifests just won't work outside of a very specific environment..) [07:37:09] the game [07:37:17] !accountreq [07:37:17] in case you want to have an account on labs, please contact someone who is in charge of doing that: Ryan.Lane, m.utante or ssmolle.tt [07:52:06] hi [08:38:03] RECOVERY dpkg-check is now: OK on wikidata-dev-1 i-00000209 output: All packages OK [08:51:03] PROBLEM dpkg-check is now: CRITICAL on wikidata-dev-1 i-00000209 output: DPKG CRITICAL dpkg reports broken packages [09:18:53] PROBLEM Puppet freshness is now: CRITICAL on wikidata-dev-2 i-0000020a output: Puppet has not run in last 20 hours [09:51:22] PROBLEM Disk Space is now: CRITICAL on deployment-transcoding i-00000105 output: DISK CRITICAL - free space: / 31 MB (2% inode=53%): [09:56:14] PROBLEM Disk Space is now: WARNING on deployment-transcoding i-00000105 output: DISK WARNING - free space: / 62 MB (4% inode=53%): [10:12:46] RECOVERY HTTP is now: OK on wikidata-dev-2 i-0000020a output: HTTP OK: HTTP/1.1 200 OK - 453 bytes in 0.004 second response time [10:40:14] PROBLEM Free ram is now: WARNING on mobile-feeds i-000000c1 output: Warning: 17% free memory [10:55:20] hi [11:15:14] PROBLEM Free ram is now: CRITICAL on mobile-feeds i-000000c1 output: Critical: 5% free memory [11:30:14] PROBLEM Free ram is now: WARNING on mobile-feeds i-000000c1 output: Warning: 7% free memory [12:13:44] PROBLEM Current Load is now: CRITICAL on wikidata-dev-1 i-0000020c output: Connection refused by host [12:14:27] PROBLEM Current Users is now: CRITICAL on wikidata-dev-1 i-0000020c output: Connection refused by host [12:15:04] PROBLEM Disk Space is now: CRITICAL on wikidata-dev-1 i-0000020c output: Connection refused by host [12:15:44] PROBLEM Free ram is now: CRITICAL on wikidata-dev-1 i-0000020c output: Connection refused by host [12:16:54] PROBLEM Total Processes is now: CRITICAL on wikidata-dev-1 i-0000020c output: Connection refused by host [12:17:34] PROBLEM dpkg-check is now: CRITICAL on wikidata-dev-1 i-0000020c output: Connection refused by host [12:21:40] hi [13:35:14] PROBLEM Free ram is now: CRITICAL on mobile-feeds i-000000c1 output: Critical: 4% free memory [14:37:04] petan: hi, do you have 10 min to talk? [14:37:44] 10min is a lot of time [14:53:34] petan: petan|wk ping ^ [14:53:43] hi [14:53:58] yes [14:54:14] PROBLEM Puppet freshness is now: CRITICAL on nova-production1 i-0000007b output: Puppet has not run in last 20 hours [14:54:15] * sumanah PMs [14:58:14] PROBLEM Puppet freshness is now: CRITICAL on nova-gsoc1 i-000001de output: Puppet has not run in last 20 hours [14:59:44] petan, anything interesting? [15:34:47] hi Platonides [16:12:00] hi petan|wk [16:45:14] PROBLEM Free ram is now: WARNING on mobile-feeds i-000000c1 output: Warning: 8% free memory [16:50:14] PROBLEM Free ram is now: CRITICAL on mobile-feeds i-000000c1 output: Critical: 4% free memory [16:55:14] PROBLEM Free ram is now: WARNING on mobile-feeds i-000000c1 output: Warning: 6% free memory [17:03:39] hi guys [17:04:03] I'm trying to install git on my windows PC using the instructions at https://labsconsole.wikimedia.org/wiki/Git#Windows but I got an error, "error: couldn't update submodule git! INSTALLATION ABORTED." [17:05:01] how do I restart the installation from where it left off? or do I need to delete the git folder in my program files and start again? [17:19:29] * THO|Cloud is back on this connection [17:19:45] I get a "your build failed... please fix it and give feedback on the git list" [17:19:58] how can I start the installation again? just re-run the installer? [17:32:21] anyone? same error I did a fresh install [17:33:46] THO|Cloud, no idea [17:33:54] complain to git folks? [17:53:47] THO|Cloud: #git [18:01:13] 04/18/2012 - 18:01:13 - Updating keys for otto [18:35:13] PROBLEM Free ram is now: CRITICAL on mobile-feeds i-000000c1 output: Critical: 4% free memory [19:00:15] PROBLEM Current Load is now: WARNING on bots-3 i-000000e5 output: WARNING - load average: 11.82, 11.10, 7.03 [19:00:15] PROBLEM dpkg-check is now: CRITICAL on bots-3 i-000000e5 output: CHECK_NRPE: Socket timeout after 10 seconds. [19:05:05] RECOVERY dpkg-check is now: OK on bots-3 i-000000e5 output: All packages OK [19:19:35] PROBLEM Puppet freshness is now: CRITICAL on wikidata-dev-2 i-0000020a output: Puppet has not run in last 20 hours [19:20:05] RECOVERY Current Load is now: OK on bots-3 i-000000e5 output: OK - load average: 3.36, 3.42, 4.63 [19:30:36] auuugh uh oh [19:30:38] hi guys [19:30:46] on one of my labs instances [19:30:47] i did: [19:30:56] sudo chmod 770 /home/otto [19:31:00] before it was 700 [19:31:29] and now [19:31:35] $ cd [19:31:35] -bash: cd: /home/otto: Permission denied [19:33:05] which instance ottomata? [19:33:09] reportcard2 [19:33:48] which project is that? I probably don't have access to it but I'll see who I can find for you [19:33:52] analytics [19:34:42] or [19:34:44] reportcard project? [19:34:52] yeah, reportcard [19:35:03] i can still sudo [19:35:08] and i've chmod-ed i back to 700 [19:35:16] but it is still pretty unhappy [19:36:10] ah i think i just fixed it [19:36:14] hmmm, not sure [19:37:59] i can cd back to my homedir now [19:38:02] but ssh is not working [19:38:07] ok going to poke at it some more [19:38:58] yes, fixed. [19:46:35] PROBLEM Free ram is now: WARNING on mobile-feeds i-000000c1 output: Warning: 6% free memory [20:01:35] PROBLEM Free ram is now: CRITICAL on mobile-feeds i-000000c1 output: Critical: 4% free memory [20:46:40] PROBLEM Disk Space is now: CRITICAL on bz-dev i-000001db output: DISK CRITICAL - free space: / 35 MB (2% inode=43%): [20:51:40] PROBLEM Disk Space is now: WARNING on bz-dev i-000001db output: DISK WARNING - free space: / 54 MB (4% inode=43%): [21:15:45] http://news.slashdot.org/story/12/04/18/0349212/wikimedia-treats-their-operations-like-their-projects [21:15:51] apparently we made slashdot too [21:24:15] <^demon> Hope we don't get slashdotted ;-) [21:51:52] Did we make hackernews yet? [22:09:20] paravoid: so, I went to the red dwarf session [22:09:39] and decided it's another PaaS that doesn't interest me in the least [22:10:07] as I was sitting there, I thought "this is just orchestration" [22:10:19] then thought, we can easily enough do this with just nova and puppet [22:10:43] I'm going to set up the two database servers as compute nodes [22:11:33] Ryan_Lane: your use of the word "orchestration" confuses me [22:11:34] * DamianZ throws sweeties at Ryan_Lane [22:11:55] http://en.wikipedia.org/wiki/Orchestration_%28computing%29 [22:12:20] Oh, it's a special term [22:12:23] yep [22:13:14] so, going to set up the two database servers as compute nodes, but they'll use LXC, rather than KVM [22:14:54] then, I'll add "predefined types" to OpenStackManager, where you can save the configuration of an instance and re-use it later [22:15:10] like puppet groups, I'll make it global and per-project [22:15:58] so, when people want a database, they'll create an instance which will be predefined as a mysql server [22:16:34] the puppet manifest will install mysql, generate a random password for mysql root, and stick that into /root/.my.cf [22:17:05] then people will be able to log into the instance and manage mysql by hand for adding other users and such [22:18:03] paravoid: heh. I'm guessing you aren't actually reading IRC [22:43:43] Ryan_Lane: sorry, I was chatting with CT [22:43:56] what's the point of doing that? [22:44:04] except of "why not" that is? :) [22:44:33] people want databases [22:44:39] they're using instances for it right now [22:44:42] and performance sucks [22:45:23] also, doing it this way means databases have some form of quota, which is the size of the instance [22:45:27] I want oauth and puppet branches too [22:45:29] :D [22:45:40] DamianZ: paravoid is working on puppet branches now :) [22:45:52] * DamianZ finds paravoid some cookies [22:45:53] I think our new security person is working on oauth [22:45:57] maybe [22:45:59] if we're lucky [22:46:13] DamianZ: liar [22:46:31] oauth or something like it is a major blocker for stuff I want to do :( puppet branches just make me not care about puppetizing stuff. [22:46:59] paravoid: You're only running http, I only like https cookies. ALL YOUR FAULT NO COOKIES [22:47:53] well, ops isn't really the right team to work on oauth [22:47:57] so, we need to beg and plead the devs [22:49:33] Devs cause all the issues :P [22:50:25] <^demon|away> Users cause all the issues when they ask for features ;-) [22:50:44] If a user has to ask for a feature the dev sucks for not writing awesome software :D [22:51:16] oauth is something I really can't believe we don't have [22:51:30] <^demon|away> Users should make do with what they're given and stop being so greedy ;-) [22:51:43] lets rewrite mediawiki in django :D [22:52:04] <^demon|away> I'll have to find a new project :( [22:52:57] So tempting to just auth people against the api but people would yell at me lots :( [22:55:53] you would still have passwords going through your code