[01:14:29] hola [01:41:51] hola nuria [01:42:05] hola [01:42:16] ok, i guess my irc client is working [01:42:27] Yup :-) Think so too. [01:42:43] But typically the channel is rather silent around such times. [01:43:14] I was just going home so will be here tomorrow [01:43:28] Sure :-) [03:18:15] (PS1) QChris: Migrate map-world_countries from yaml to json [analytics/global-dev/dashboard-data] - https://gerrit.wikimedia.org/r/100737 [03:18:30] (PS1) QChris: Adjust thresholds to pass at least for local checkouts again [analytics/geowiki] - https://gerrit.wikimedia.org/r/100738 [03:18:31] (PS1) QChris: Fetch map-world_countries as json [analytics/geowiki] - https://gerrit.wikimedia.org/r/100739 [03:18:32] (PS1) QChris: Split files into public and private [analytics/geowiki] - https://gerrit.wikimedia.org/r/100740 [03:18:33] (PS1) QChris: Give proper error message when fetching of URL failed [analytics/geowiki] - https://gerrit.wikimedia.org/r/100741 [03:18:34] (PS1) QChris: Do not check certificates when fetching files [analytics/geowiki] - https://gerrit.wikimedia.org/r/100742 [03:18:35] (PS1) QChris: Use different URLs and authentication for private data [analytics/geowiki] - https://gerrit.wikimedia.org/r/100743 [03:18:36] (PS1) QChris: Allow to request checking limn scaffolding for private data [analytics/geowiki] - https://gerrit.wikimedia.org/r/100744 [03:28:55] ottomata: Are you still around? [03:29:00] yup! [03:29:17] Awesome! I've got a quick question about puppet and passwords [03:29:30] yup [03:29:31] ask away [03:29:50] For the geowiki monitoring, I want to use the http password for the stats user in a file. [03:30:13] Is it better to put the password in a puppet variable and write the password file through puppet [03:30:46] Or do you prefer to have the password file in the puppet:///private part? [03:34:13] ah [03:34:33] wait what's the first option? [03:34:46] oh [03:34:48] i see [03:34:52] i think the first option i sbetter [03:34:57] its more portable [03:34:59] Ok. [03:35:07] so, puppet variable [03:35:10] And what do I have to do to get it into a variable? [03:35:14] and the password template file is in regular puppet [03:35:16] i have to do it for you [03:35:19] or someone in ops [03:35:20] but [03:35:23] probably [03:35:24] something like [03:35:57] include passwords::geowiki [03:35:57] $geowiki_http_user = $passwords::geowiki::user [03:35:57] …_pass = ::pass [03:35:59] something like that [03:36:24] ok. I'll give it a shot. [03:36:27] Thanks! [03:36:51] yup! [03:56:11] qchris: i'm around for merging stuff right now, you wanna? [03:56:25] better not. [03:56:43] It depends on some changes in the monitoring script that need review. [03:56:52] ok [03:56:53] ottomata: Or do you want to review thath as well? [03:56:57] yeah i can [03:56:58] Its just bash ... [03:57:07] ja sure [03:57:08] https://gerrit.wikimedia.org/r/#/c/100743/ [03:57:13] add me as reviewers on these [03:57:16] and all its ancestors [03:57:18] ok. [03:57:26] having trouble finding them from just IRC in #ops room [03:58:41] ottomata: Done [04:01:07] (CR) Ottomata: [C: 2] Adjust thresholds to pass at least for local checkouts again (1 comment) [analytics/geowiki] - https://gerrit.wikimedia.org/r/100738 (owner: QChris) [04:01:21] just a comment typo on that one qchris [04:01:26] should I Just merge and you can fix later? [04:01:58] No just bug me right away :-) [04:02:55] Whatever you prefer. [04:03:09] ahh i'll just merge :) [04:03:13] you can fix in some future commit [04:03:19] (CR) Ottomata: [V: 2] Adjust thresholds to pass at least for local checkouts again [analytics/geowiki] - https://gerrit.wikimedia.org/r/100738 (owner: QChris) [04:03:19] Ok. [04:03:51] (CR) Ottomata: [C: 2 V: 2] Fetch map-world_countries as json [analytics/geowiki] - https://gerrit.wikimedia.org/r/100739 (owner: QChris) [04:04:18] Ah. That's not a typo on our end. That's what the column is called. :-((( [04:04:41] hah [04:04:42] really? [04:04:46] Unkowns? [04:04:56] We could fix in the generating script. But that might break things for other consumers [04:05:02] So I never fixed it :-) [04:05:10] crazy [04:05:11] ha [04:05:11] ok [04:05:19] Just grep for Unkown in the file :-) [04:05:30] (CR) Ottomata: [C: 2 V: 2] Split files into public and private [analytics/geowiki] - https://gerrit.wikimedia.org/r/100740 (owner: QChris) [04:06:01] (CR) Ottomata: [C: 2 V: 2] Give proper error message when fetching of URL failed [analytics/geowiki] - https://gerrit.wikimedia.org/r/100741 (owner: QChris) [04:06:30] (CR) Ottomata: [C: 2 V: 2] Do not check certificates when fetching files [analytics/geowiki] - https://gerrit.wikimedia.org/r/100742 (owner: QChris) [04:09:46] (CR) Ottomata: [C: 2 V: 2] Use different URLs and authentication for private data [analytics/geowiki] - https://gerrit.wikimedia.org/r/100743 (owner: QChris) [04:10:53] Awesome. Thanks! [04:11:29] qchris, i don't understand this one [04:11:29] 100744 [04:11:31] https://gerrit.wikimedia.org/r/#/c/100744/ [04:12:37] So we split generating into a private and a public repository [04:12:49] Both generate all the limn graphs, dashboards etc. [04:13:11] But for the private repository, the are not available through a web server. [04:13:18] So we need not test them. [04:13:42] But you might want to test them if you have a checkout of the private repository on your local disk [04:13:59] So that change adds a parameter to allow just that. [04:14:21] ahh i see [04:14:25] (This mode makes my life easier when developing/testing geowiki) [04:14:28] ok so the cron won't check by default because it can't [04:14:33] but if you happen to have and want to verify them [04:14:34] Correct. [04:14:34] you can [04:14:37] cool [04:14:44] That change is not needed by the cron. [04:14:46] (CR) Ottomata: [C: 2 V: 2] Allow to request checking limn scaffolding for private data [analytics/geowiki] - https://gerrit.wikimedia.org/r/100744 (owner: QChris) [04:16:43] hmm, qchris, [04:16:52] yes? [04:16:59] how is the http auth for the geowiki url getting set up right now? [04:17:16] You mean the part on stat1001? [04:17:22] Through htpasswd. [04:17:37] yeah, but is it in puppet? or was that manually set up (by me?) [04:17:46] looking.. [04:17:53] We hand-edit the htpasswd file. [04:18:07] And then I ask you to upload it again. [04:18:12] ahhhh [04:18:13] yes [04:18:15] ok hm [04:18:41] The stats user is already available in the htpasswd on stat1001. [04:18:57] So no need to change anything there. [04:19:57] ahh ok, i see [04:19:59] ok cool right [04:20:25] I sent you an email where to find the plaintext http password for the stats user [04:20:27] hashed pw too, ok ok ok, i was going to suggest that we use the same variable from passwords class to render both htpasswd file and this cron, buuut, that's ok [04:20:27] yeah [04:20:29] got that [04:20:37] but since they are actually different content, meh [04:21:55] i'll add a comment in passwords.pp that if that pass is changed, it should also be changed in "puppet:///private/apache/htpasswd.stats [04:22:07] Awesome. [04:26:39] qchris [04:26:40] is [04:26:46] $geowiki_base_path set in this scope? [04:27:24] https://gerrit.wikimedia.org/r/#/c/100747/2/manifests/misc/statistics.pp [04:27:28] line 1060 [04:28:05] I guess not :-( [04:28:08] Let me check again. [04:28:59] I updated the change. [04:34:00] looking gooooOood [04:34:07] you want to try to run the command on stat1? [04:34:14] oh [04:34:18] i have to run it right? [04:34:20] sudo -u stats? [04:34:34] qchris ^ [04:34:40] Yes [04:34:47] Trying the cronscript right now. [04:34:52] you can do it? [04:35:29] Fails to read a line from the password file [04:37:34] hm [04:37:55] possibly because there is no newline? doubtful. [04:41:25] (PS1) QChris: Do not require line ending in http password file [analytics/geowiki] - https://gerrit.wikimedia.org/r/100751 [04:41:59] ottomata: Ja. That was it. [04:42:09] (CR) Ottomata: [C: 2 V: 2] Do not require line ending in http password file [analytics/geowiki] - https://gerrit.wikimedia.org/r/100751 (owner: QChris) [04:42:13] kj [04:42:21] looks good [04:42:25] try again? [04:43:25] Doing that ... hod to override date as UTC switched dates already. [04:43:30] s/hod/had/ [04:43:45] Ok, now the monitoring has access :-) [04:43:49] woot [04:43:51] Yippie! [04:43:55] Thanks a lot ottomata [04:44:37] Now off to bed :-) [04:45:33] Good night. [04:46:01] me too [04:46:02] niighters [14:57:46] hey qchris / ottomata / average / tnegrin [14:57:51] wanna have standup at 9 PST? [14:57:56] that is 2 hours from now [14:58:08] hah, someone is on the west coast! [14:58:11] Ok. [14:58:15] 17:00 utc [14:58:23] sure, fine with meeee [14:58:23] i'm not there yet [14:58:26] oh [14:58:26] but nuria is [14:58:35] mk [14:58:38] cool [14:59:09] k, i'll jump into the standup to tell toby and stefan [15:02:38] ottomata: aah, changed my mind again [15:02:41] standup now? [15:02:45] seems to work better for evryone [15:38:49] milimetric: what about wikimetrics mysql db [15:38:54] does that need puppetized / installed? [15:39:00] should it be localhost always? [15:39:25] it could be anywhere, and two things need to happen before wikimetrics can use it [15:39:28] 1. create the database [15:39:37] 2. create the wikimetrics user with the password in the config [15:39:55] do you have a script in wikimetrics repo to create db? [15:39:56] I have, for the life of me, not been able to do 2. automatically [15:39:59] i can make a wikimetrics::databse class [15:40:08] like, my sql's "set password" won't work for me unless I do it manually [15:40:11] and others have reported the same [15:40:17] i always use grant [15:40:31] grant doesn't set passwords though right? [15:40:41] identified by [15:40:42] sure [15:40:46] you can do hashed pws too [15:40:51] identified by password '..sdfsdfdh' [15:41:08] i have this line "set password for 'wikimetrics'@'%' = PASSWORD('blah') [15:41:11] and it like never works in a script [15:41:23] check this [15:41:23] https://github.com/wikimedia/puppet-cdh4/blob/master/manifests/oozie/database/mysql.pp#L38 [15:41:59] that's a seemingly crazy side-effect of grant... to create the user [15:42:00] but ok [15:42:01] ! :) [15:42:04] oo even this [15:42:04] https://github.com/wikimedia/puppet-cdh4/blob/master/manifests/hive/metastore/mysql.pp#L49 [15:42:24] i think i tried the create user syntax [15:42:26] that didn't seem to work [15:42:33] anyway, who knows what horrible thing I did [15:42:39] heh, ja i thikn i can do [15:43:49] cool, you da man [15:44:14] hm [15:44:27] i notice that there are nowMETA_MW_CLIENT_SECRET and the like in web_config [15:44:30] do we need those puppetized? [15:44:41] and twitter is gone now? :) [15:45:13] oh [15:45:19] yes, sorry [15:45:27] meta mediawiki [15:45:36] it's the OAuth setup, and Twitter was not gonna happen so I took it out [15:45:53] ok cool [15:46:02] that just happened last week after I made the first config versions [15:46:16] I can update the configs? [15:46:49] or are you in the middle of a commit? [15:47:04] i can do it [15:47:06] i'm working on it now anyway [15:54:16] milimetric: do tables need to be created? [15:54:30] no, sqlalchemy will do that if they don't exist [15:54:59] ottomata: shouldn't this file use $config_directory? https://gerrit.wikimedia.org/r/#/c/96042/12/modules/wikimetrics/templates/upstart.wikimetricsqueue.conf.erb [15:55:21] yup, good catch [16:02:22] hmm, milimetric [16:02:26] from apache [16:02:26] IOError: [Errno 2] No such file or directory: '/etc/wikimetrics/queue_config.yaml' [16:02:36] i thought web wouldn't need queue config? [16:02:44] it shouldn't [16:03:02] hm, that's weird [16:03:24] https://gist.github.com/ottomata/7913060 [16:04:56] oh i see [16:05:07] that makes sense but it's a bug [16:05:12] hard to fix... hm... [16:08:03] yeah ottomata, can we just include all the configs for all types for now? [16:08:20] booooo :) [16:08:31] queue won't run [16:08:34] but it's needed by web [16:08:37] to make the bridge to celery [16:09:02] and one stupid flask thing is needed by celery [16:09:09] and i'm not sure how to decouple them right now [16:09:11] celery needs web_config? [16:09:17] but they just need to be configured [16:09:19] well, no [16:09:26] celery needs to import something from app [16:09:33] but for app to exist, you must feed it configuration [16:09:49] uhhh, hm ok [16:10:26] so yea, it turns out it's much easier if all the config files are all there [16:10:47] basically, at import time, there's no way to know what the parser args were [16:10:53] because it happens before run time [16:54:27] oook milimetric [16:54:27] https://ottometrics.wmflabs.org/ [16:54:29] that's looking better [16:54:30] :) [16:54:55] ooh, fancy :) [16:55:15] if you tail /var/log/upstart/wikimetricsqueue.log, do you see the queue being up and happy? [16:55:33] don't have queue running yet [16:55:36] k [16:55:38] need to talk to you about how to do that [16:55:41] sure [16:55:42] hangout? [16:55:42] what needs to be done for redis [16:55:43] ? [16:56:00] redis hmmm [16:56:08] lemme grab the config I have in labs [16:56:43] hm, i can probably just include the redis module [17:01:42] ottomata: here's the config without comments: https://gist.github.com/milimetric/7914347 [17:01:57] the only things I *think* I want to change are: [17:02:04] demonize yes [17:02:25] save 900 1 [17:02:25] save 300 10 [17:02:25] save 60 20 [17:03:00] the rdb dump settings: [17:03:00] dbfilename dump.rdb [17:03:00] dir /var/lib/redis/ [17:03:08] (I'm not exactly sure what I need there) [17:03:46] and I think that's it [17:03:58] the rest I think are defaults [17:04:33] can dbfilename be <%= hostname %>-<%= port %>.rdb [17:04:34] ? [17:04:40] that's what it is in redis module [17:04:43] oh that's right, I had a problem with dump.rdb not being written to because I hadn't started redis with a user that had rights to it [17:04:49] sure [17:04:54] hostname-port works [17:05:33] hmm also [17:05:36] save 900 1 [17:05:36] save 300 100 [17:05:36] save 60 10000 [17:05:37] because wikimetrics is not super high-traffic I tweaked the default save lines [17:05:38] is what puppet module wants [17:05:44] i know, i'd rather not use that [17:05:47] hm [17:05:50] it would mean not saving very often [17:05:52] grr puppet module is not very flexible [17:05:55] yea [17:06:05] hmmi can do it [17:06:13] :) I know you can [17:06:22] thou art the puppet master [17:07:06] at the office [17:09:24] oh hey nuria [17:09:37] hello [17:09:57] i was going to get started with this: https://github.com/wikimedia/limn#install [17:10:10] well, before you do that, let's get that deck together that Toby wanted [17:10:20] ok, [17:10:28] he needs it in 2 hours and it'll be a good opportunity to walk you through some high level stuff we do [17:10:42] very well [17:18:36] milimetric: https://gist.github.com/ottomata/7914588 [17:18:36] ? [17:18:40] from queue [17:19:23] oh i see many more configs on wikimetrics instane in queue_config now [17:19:23] adding [17:22:54] ok mostly working, except now [17:22:55] [2013-12-11 17:22:45,796: ERROR/MainProcess] Consumer: Connection Error: [Errno 111] Connection refused. Trying again in 14 seconds... [17:23:25] ok, going to get some lunch, back in a little while [18:39:28] later everyone [18:39:40] i'm off to the airport [18:45:31] laaters! [19:33:12] ottomata, average are we having retrospective today (I feel lonely in both batcave and event hangout)? [19:34:38] i just joined it [19:34:40] dunno! [19:34:44] dan is flying, so probalby not [19:36:10] Ah the batcave!