[00:28:57] drdee: Hey you're doing API stats, right? Could you get statistics for which values of the &format= parameters are used how often? [00:30:55] petan: you about? [00:31:13] I opened http://labs.wikimedia.beta.wmflabs.org/wiki/Special:RecentChanges without being logged in (which is strange as I thought that I saved my log in) [00:31:22] and it told me last RC was on 19 January 2012 [00:38:23] drdee: Additionally, I'd like to know if action=userdailycontribs is actually used by people [02:38:15] RECOVERY Free ram is now: OK on bots-sql3 bots-sql3 output: OK: 20% free memory [02:40:55] PROBLEM Free ram is now: WARNING on puppet-lucid puppet-lucid output: Warning: 12% free memory [02:41:45] RECOVERY Free ram is now: OK on deployment-backup deployment-backup output: OK: 23% free memory [02:54:45] PROBLEM Free ram is now: WARNING on deployment-backup deployment-backup output: Warning: 19% free memory [02:56:15] PROBLEM Free ram is now: WARNING on bots-sql3 bots-sql3 output: Warning: 18% free memory [03:00:55] PROBLEM Free ram is now: CRITICAL on puppet-lucid puppet-lucid output: Critical: 3% free memory [03:30:55] RECOVERY Free ram is now: OK on puppet-lucid puppet-lucid output: OK: 20% free memory [03:34:55] PROBLEM host: nova-test1 is DOWN address: nova-test1 check_ping: Invalid hostname/address - nova-test1 [08:35:30] hi all [08:39:36] RoanKattouw: hi [08:39:43] need any help on deployment site? [08:40:40] No, I'm good [08:40:54] It took me a while to figure out how to edit the settings but I managed to do it last night [08:41:13] And I had to change the SVN checkout so that you could run svn up without permissions errors if your username wasn't mah [08:41:32] But all of the bugs I was investigating I either fixed or just went away on their own [08:48:17] RoanKattouw: can you update squid? [08:48:22] you have access to prod version [08:48:29] it's in deployment-squid [08:48:35] ? [08:48:42] in /etc/squid [08:48:52] it's configured comepletely different than pro [08:48:53] I managed to purge URLs from Squid using MediaWiki's scripts, if that's what you mean [08:48:55] Oh [08:49:06] Yeah, I have the config, but I'm still waiting on Ryan to check it [08:49:19] afaik it's going to take a long time [08:49:25] because he wants to move it to puppet [08:49:30] No, not realy [08:49:36] We said we weren't gonna puppetize Squid [08:49:37] I just need to update the config strings, not private data [08:49:43] ok [08:49:44] Exactly because it's not worth the effort [08:50:16] actually config with no ip addresses is ok [08:50:16] Instead, we're gonna do a minimal puppetization, i.e. we'll still have PHP scripts that generate config files like we have now, we're just gonna use puppet to place the generated files on the servers [08:50:17] for me [08:50:25] ok [08:50:35] problem is that squid now causes a lot of troubles [08:50:41] Is the Squid IP configured in MediaWiki too? [08:50:49] if Ryan was online I would ask him to increase the quota on deployment [08:50:53] and disabled squid at all [08:51:04] yes it is [08:51:11] (Yeah, I noticed that a lot of the bugs I was looking at seemed to magically go away if I purged a URL in Squid) [08:51:20] Quota for what? Disk space? Memory? [08:51:27] number of vm's [08:51:30] we have 10 vm's [08:51:35] I need to create more web servers [08:51:43] when I turn off squid cache, they are going to be loaded a lot [08:52:10] now we have only 2 servers [08:52:12] for apache [08:52:25] btw could you set up ssl [08:52:29] on 443 [08:52:32] :o [08:52:37] I have no idea how to do that [08:52:38] Oh, right [08:52:57] Actually, I would like to do that with an nginx proxy [08:53:02] But I can't because I'd need another VM :( [08:53:11] I created a vm for ssl [08:53:13] (Production also uses nginx proxies for SSL) [08:53:18] oh!' [08:53:20] it's called deployment-webs1 [08:53:25] try it [08:54:58] it has running web server there [08:55:21] It the web server used? [08:55:25] no [08:55:38] I was just testing it [08:55:44] OK [08:56:24] Oh, look at that [08:56:43] Ryan was nice enough to make certificates::star_wmflabs_org available [08:57:01] Now I just need to figure out how it differs from certificates::star_wmflabs (without the org) :S [08:57:33] heh [08:57:36] It's... not? WTF [08:57:37] ok [08:59:28] OK seriously [08:59:41] There's star.wmflabs.pem and star.wmflabs.org.pem , and they're different [08:59:48] But guess what, neither is being used [08:59:53] Both classes actually install wmf-labs.pem [09:00:25] Aah, the other two are being used elsewhere [09:07:05] :o [09:09:25] It's weird [09:09:38] But it should be possible to set up an nginx SSL proxy this way [09:10:57] ok [09:11:04] you will need to set up a squid for it too [09:11:14] because all traffic is first going through it [09:11:24] No, it's the other way around [09:11:29] First the SSL gateway, then Squid [09:11:34] * RoanKattouw sighs [09:11:53] ok, so I have to re IP [09:11:59] Of course the SSL proxy puppet class is not at all reusable, it's got WMF IPs hardcoded all over [09:12:00] move it [09:12:07] Well, eventually yes [09:12:09] eh [09:12:10] ok [09:12:20] but first we need to get it working [09:12:22] I'm not really sure how this works tbh [09:12:26] :) [09:12:32] someone know [09:12:34] ? [09:12:44] I guess some folks from ops should do [09:12:50] Because on the cluster port 80 traffic goes to the Squids directly (AFAIK), while port 443 traffic goes through nginx [09:13:05] Well, I can tell you who set up the nginx proxy, but you won't like the answer [09:13:07] ok I can nat it [09:13:14] we could set up a nat on squid [09:13:19] guess so [09:13:24] that's what I wanted to do [09:13:36] If that Just Works then it's fine [09:13:40] in that case we don't need to change much [09:13:46] Ideally I'd like this project to be as close to deployment as we can get [09:13:52] but I don't know much about the ip tables :) [09:14:07] close to deployment? [09:14:10] what you mean [09:14:46] Well, ideally I'd like to do the port 80/443 traffic splitting the same way it's being done in prod. Probably involves LVS [09:14:55] But that's hard, especially with Ryan on vacation [09:14:55] ah [09:15:08] yes, but mutante can do same as Ryan [09:15:12] Also, the nginx puppet manifest isn't the most reusable thing ever and would need to be refactored [09:15:24] True, someone else might know how this works [09:15:27] Mark probably does [09:15:32] ok [09:15:58] * RoanKattouw wonders why he's still talking about something he's not gonna have time to do anywayy [09:16:08] heh [09:17:07] @search stat [09:17:07] Results (found 2): monitor, osm-bug, [09:17:31] @regsearch ^g.* [09:17:32] Results (found 2): password, leslie's-reset, [09:17:42] @regsearch \sg.* [09:17:42] Results (found 8): git, damianz's-reset, sudo, ghsh, newgrp, ryanland, account-questions, account, [09:17:53] !ghsh [09:17:53] $1 $2 gsdfgs$34ghrsh $4gsh _$5 [09:17:59] o.o [09:18:19] @search gan [09:18:19] Results (found 2): load, load-all, [09:18:23] !load-all [09:18:23] http://ganglia.wikimedia.org/2.2.0/?c=Virtualization%20cluster%20pmtpa&m=load_one&r=hour&s=by%20name&hc=4&mc=2 [09:18:30] !load [09:18:30] http://ganglia.wikimedia.org/2.2.0/graph_all_periods.php?h=virt2.pmtpa.wmnet&m=load_one&r=hour&s=by%20name&hc=4&mc=2&st=1327006829&g=load_report&z=large&c=Virtualization%20cluster%20pmtpa [09:30:40] RoanKattouw: what he's on vacation? [09:30:50] damn [09:32:08] Let me see when he gets back [09:32:27] Monday [09:32:31] eh... ok [09:32:50] (that is, he'll probably get home over the weekend or something but Monday is his first day back in the office) [09:32:52] btw is he in europe? I saw him online on gtalk in my time [09:32:57] was confused [09:32:58] Yeah [09:33:03] aha [09:33:08] He presented about labs at FOSDEM in Brussels [09:33:13] yay [09:33:23] that's not so far [09:33:54] There will be video on line later, apparently [10:02:43] Heh, Feb 18 at 18:00 UTC [10:02:54] * RoanKattouw may well be on a plane then [10:03:08] (but that's fine; I would like to be there but I can read the logs later too) [10:12:30] 18:00 GMT [10:12:32] not UTC [10:12:39] or is it same? :P [10:12:45] I don't this :D [10:12:47] heh [10:12:49] doodle ftw [10:12:52] :) [10:14:36] !labsconf | RoanKattouw [10:14:36] RoanKattouw : http://etherpad.wikimedia.org/LabsIrcConf [10:14:48] feel free to update it [10:15:12] GMT == UTC, modulo leap seconds but no one cares about those really [10:15:35] The conference will happen on 18. Feb. at 22:00 GMT [10:15:37] ? [10:15:48] yay [10:15:54] :) [10:15:57] that was old date [10:16:02] heh [10:16:04] actually I think we could change the time [10:16:09] Oh you fixed it already [10:16:15] I think it was decided on doodle [10:16:17] Well, I don't care in particular [10:16:20] most of people voted for this time [10:16:27] actually I don't care either [10:17:02] I probably won't be there anyway, unless maybe I get to fly Lufthansa and their in-flight wifi thing actually works [10:17:23] :) [10:17:47] maybe Ryan tell me he won't be here anyway and I will have to change it heh [10:18:02] And I guess Ryan is gonna have to abandon his principles and actually come into the office before 10 for once [10:18:06] (18:00 UTC = 10:00 PST) [10:18:13] it's saturday [10:18:17] no need to be in office [10:18:47] that's why most of people voted for saturday [10:18:56] including Ryan [10:19:04] but he voted for later time :) [10:19:11] D'oh, right [10:19:12] Saturday [14:14:19] ACKNOWLEDGEMENT Free ram is now: WARNING on bots-sql3 bots-sql3 output: Warning: 18% free memory [14:35:56] New patchset: Dzahn; "planet - update script,cron,systemuser,logdir,.." [operations/puppet] (test) - https://gerrit.wikimedia.org/r/2441 [14:36:14] New review: gerrit2; "Change did not pass lint check. You will need to send an amended patchset for this (see: https://lab..." [operations/puppet] (test); V: -1 - https://gerrit.wikimedia.org/r/2441 [14:37:51] New patchset: Dzahn; "planet - update script,cron,systemuser,logdir,.." [operations/puppet] (test) - https://gerrit.wikimedia.org/r/2441 [14:38:09] New review: gerrit2; "Lint check passed." [operations/puppet] (test); V: 1 - https://gerrit.wikimedia.org/r/2441 [14:38:34] New review: Dzahn; "(no comment)" [operations/puppet] (test); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2441 [14:38:34] Change merged: Dzahn; [operations/puppet] (test) - https://gerrit.wikimedia.org/r/2441 [14:45:43] err: Could not retrieve catalog from remote server: Error 400 on SERVER: Could not find class generic::webserver::php5 for i-000000ea.pmtpa.wmflabs on node i-000000ea.pmtpa.wmflabs [14:46:15] hmm, whether i try to use it from global puppet group or add the class to my project, it cant find it anymore, currently [14:48:48] just webserver::php5 now when looking at others [14:58:50] New patchset: Dzahn; "require user and script for cronjob" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/2445 [14:59:10] New review: gerrit2; "Lint check passed." [operations/puppet] (test); V: 1 - https://gerrit.wikimedia.org/r/2445 [14:59:18] New review: Dzahn; "(no comment)" [operations/puppet] (test); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2445 [14:59:18] Change merged: Dzahn; [operations/puppet] (test) - https://gerrit.wikimedia.org/r/2445 [15:30:55] * jeremyb pokes Ryan_Lane [15:31:03] jeremyb: [15:31:04] ? [15:31:07] hey [15:31:10] still on the road? [15:31:38] * jeremyb has been missing you (bad TZ) for a few days [15:31:51] * jeremyb gets link [15:32:02] Ryan_Lane: https://labsconsole.wikimedia.org/wiki/Requests [15:32:32] jeremyb: I'm on vacation. in europe, so haven't been around. [15:33:06] ahh, didn't know (i knew FOSDEM but i thought maybe then pune?) [15:33:19] hence i asked about still on the road ;) [15:33:43] Ryan_Lane: anyway, it's (at least the part i'm poking about now) just a project creation [15:33:57] idk if there's anyone else to do them while you're gone [15:34:06] anyone on ops can do them [15:34:19] just have to find a person who knows how. heh [15:34:25] i know technically they can but i thought you didn't want them too [15:36:07] Ryan_Lane [15:36:20] nah. I don't mind anyone else doing it [15:36:26] k [15:37:52] Ryan_Lane: Hope you where in a certain place called brussels last weekend :) [15:38:03] Damianz: I was [15:38:10] Damianz: talked at fosdem [15:38:13] Sweet [15:38:18] packed the 500 person room [15:38:25] I couldn't make it in the end so I'm having to wait for the videos :( [15:38:53] Though I've still got your wmfhackathon video to watch :P [15:40:09] Ryan_Lane: when do you return? [15:41:06] wmfhackathon is really like a 5 minute "here's what labs can do for you here at this hackathon" [16:14:14] New patchset: Dzahn; "add all the (old) planet config.ini files to work with" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/2448 [16:25:15] hmm, user "statd" owns files that i didnt expect it to own [16:34:02] New review: Dzahn; "(no comment)" [operations/puppet] (test); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2448 [16:34:03] Change merged: Dzahn; [operations/puppet] (test) - https://gerrit.wikimedia.org/r/2448 [16:35:17] jeremyb: the 12th [16:35:23] probably [16:36:02] New patchset: Dzahn; "add planet configs using a custom definition to avoid repetition" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/2449 [16:38:13] New patchset: Dzahn; "add planet configs using a custom definition to avoid repetition, add mode" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/2449 [16:38:31] New review: gerrit2; "Lint check passed." [operations/puppet] (test); V: 1 - https://gerrit.wikimedia.org/r/2449 [16:38:47] New review: Dzahn; "(no comment)" [operations/puppet] (test); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2449 [16:38:48] Change merged: Dzahn; [operations/puppet] (test) - https://gerrit.wikimedia.org/r/2449 [16:40:47] New patchset: Dzahn; "fix typo in file path breaking dependency" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/2450 [16:41:07] New review: Dzahn; "(no comment)" [operations/puppet] (test); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2450 [16:41:07] Change merged: Dzahn; [operations/puppet] (test) - https://gerrit.wikimedia.org/r/2450 [16:41:07] New review: gerrit2; "Lint check passed." [operations/puppet] (test); V: 1 - https://gerrit.wikimedia.org/r/2450 [16:53:57] Ryan_Lane: ok, enjoy the east! [16:56:56] New patchset: Dzahn; "planet - fix output_pathes and remove Brion" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/2451 [16:57:16] New review: gerrit2; "Lint check passed." [operations/puppet] (test); V: 1 - https://gerrit.wikimedia.org/r/2451 [16:57:20] jeremyb: will do [16:58:31] New review: Dzahn; "(no comment)" [operations/puppet] (test); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2451 [16:58:32] Change merged: Dzahn; [operations/puppet] (test) - https://gerrit.wikimedia.org/r/2451 [17:08:47] New patchset: Dzahn; "using that definition didnt work as expected, (why?), then just a more direct way.." [operations/puppet] (test) - https://gerrit.wikimedia.org/r/2454 [17:09:09] New review: gerrit2; "Lint check passed." [operations/puppet] (test); V: 1 - https://gerrit.wikimedia.org/r/2454 [17:10:19] New review: Dzahn; "(no comment)" [operations/puppet] (test); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2454 [17:10:19] Change merged: Dzahn; [operations/puppet] (test) - https://gerrit.wikimedia.org/r/2454 [17:16:26] New patchset: Dzahn; "$name instead of $title, $title has the class name" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/2455 [17:16:50] New review: Dzahn; "(no comment)" [operations/puppet] (test); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2455 [17:16:50] Change merged: Dzahn; [operations/puppet] (test) - https://gerrit.wikimedia.org/r/2455 [17:16:51] New review: gerrit2; "Lint check passed." [operations/puppet] (test); V: 1 - https://gerrit.wikimedia.org/r/2455 [17:42:54] New patchset: Dzahn; "back to using definiton, but do _not_ use - in definition names, may think i want to do some maths" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/2457 [17:43:15] New review: gerrit2; "Lint check passed." [operations/puppet] (test); V: 1 - https://gerrit.wikimedia.org/r/2457 [17:43:37] New review: Dzahn; "(no comment)" [operations/puppet] (test); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2457 [17:43:37] Change merged: Dzahn; [operations/puppet] (test) - https://gerrit.wikimedia.org/r/2457 [18:02:18] New patchset: Dzahn; "ok that worked, now ensure document root dirs exist, config languages in single place, system role" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/2458 [18:02:38] New review: gerrit2; "Lint check passed." [operations/puppet] (test); V: 1 - https://gerrit.wikimedia.org/r/2458 [18:03:07] New review: Dzahn; "(no comment)" [operations/puppet] (test); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2458 [18:03:07] Change merged: Dzahn; [operations/puppet] (test) - https://gerrit.wikimedia.org/r/2458 [18:17:11] New patchset: Dzahn; "adding the "managed by puppet"-warnings to all configs" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/2461 [18:17:31] New review: gerrit2; "Lint check passed." [operations/puppet] (test); V: 1 - https://gerrit.wikimedia.org/r/2461 [18:17:45] New review: Dzahn; "(no comment)" [operations/puppet] (test); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2461 [18:17:45] Change merged: Dzahn; [operations/puppet] (test) - https://gerrit.wikimedia.org/r/2461 [18:24:44] New patchset: Dzahn; "add all planets to index.html" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/2462 [18:25:03] New review: gerrit2; "Lint check passed." [operations/puppet] (test); V: 1 - https://gerrit.wikimedia.org/r/2462 [18:25:17] New review: Dzahn; "(no comment)" [operations/puppet] (test); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2462 [18:25:18] Change merged: Dzahn; [operations/puppet] (test) - https://gerrit.wikimedia.org/r/2462 [18:33:41] New patchset: Dzahn; "that's the drawback, our old template files arent compatible and break planet runs, let all use the "musings" theme for now" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/2463 [18:34:01] New review: gerrit2; "Lint check passed." [operations/puppet] (test); V: 1 - https://gerrit.wikimedia.org/r/2463 [18:34:26] New review: Dzahn; "(no comment)" [operations/puppet] (test); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2463 [18:34:26] Change merged: Dzahn; [operations/puppet] (test) - https://gerrit.wikimedia.org/r/2463 [18:45:12] http://planet.wmflabs.org/ [21:26:20] Meeting for what? [21:27:12] Damianz: http://lists.wikimedia.org/pipermail/wikitech-l/2012-February/058005.html [21:27:17] Damianz: a variety of subjects [21:27:26] specs, legal stuff, account creation, backups, etc [21:27:58] Damianz: added link to that agenda to /topic, thanks for reminder [21:28:10] back later [21:31:40] backups? we haz those? [21:31:41] :P [22:27:12] 02/09/2012 - 22:27:12 - Creating a home directory for otto at /export/home/bastion/otto [22:28:11] 02/09/2012 - 22:28:11 - Updating keys for otto [22:29:16] 02/09/2012 - 22:29:15 - Creating a home directory for otto at /export/home/mobile-stats/otto [22:30:16] 02/09/2012 - 22:30:16 - Updating keys for otto [23:16:39] !labsconf [23:16:39] http://etherpad.wikimedia.org/LabsIrcConf [23:19:47] drdee: I'm sorry, I can't copy anything to your labs instance. I don't have access. [23:42:23] maplebed: hold on, i'll add you [23:42:59] drdee: sorry, fighting a production bug at the moment. [23:45:00] ok, i understand but you have been added [23:45:16] 02/09/2012 - 23:45:16 - Creating a home directory for ben at /export/home/mobile-stats/ben [23:46:16] 02/09/2012 - 23:46:15 - Updating keys for ben