[01:15:54] (PS1) Gergő Tisza: Add above-fold filepage link to stats fields [analytics/multimedia] - https://gerrit.wikimedia.org/r/140263 [01:19:48] (PS2) Gergő Tisza: Add above-fold filepage link to stats fields [analytics/multimedia] - https://gerrit.wikimedia.org/r/140263 [01:22:12] (PS1) Gergő Tisza: Add click tracking for the above-the-fold file page link [analytics/multimedia/config] - https://gerrit.wikimedia.org/r/140264 [07:56:32] nuria: you asked about the backup scripts and why it's bash not python. [07:56:46] I do not think that it's a strict requirement. [07:57:02] But as it's essentially copying a few files over [07:57:16] I like milimetric's decision to just use a shell script. [07:57:33] python is just an extra layer around the commands. [07:57:50] a 'friendly' extra layer you mean [07:57:55] But just because I like it, does not mean that it has to stay that way. [07:58:02] What would be the benefits of using Python [07:58:03] ? [07:58:23] No, not a "friendly", an "unnecessary" extra layer. [07:58:31] well is a lot more friendlier to do the same thing [07:58:46] i find bash scripts hard to read as i write them not so often [07:59:11] so i have to rememeber every syntax nuance(?) [07:59:37] whereus we write python every day i can read it a lot easier [08:00:17] Already starting a program in Python and properly treating errors is really hard. [08:00:27] But I am not sold on shell. [08:00:41] If you say Python is better, I'll gladly review your Python code. [08:02:05] But I would not switch languages for nothing. But if the Python code rocks, I am convinced :-) [08:02:57] For this item, no , no need from what you were saying before is almost done. Now, going forward I rather write these types of scripts in python, if possible. [08:04:04] or perl [08:04:15] I don't. I like bash. [08:04:16] but that would be kind of old fashioned maybe. [08:04:21] It's the right tool for these kind of things. [08:04:32] qchris: http://mywiki.wooledge.org/BashFAQ/105 ;) [08:04:55] ori? [08:05:12] Did I misuse "set -e" somewhere? [08:05:19] Or is it just that bash is hard too? [08:05:30] no, i'm just teasing you about error-handling in python being convoluted :P [08:05:43] hola ori [08:05:53] what 'set -e' does depends on the phase of the moon, the time you last brushed your teeth, etc [08:05:55] hey nuria [08:06:14] i was about to respond to your ping re: edge with a sad note saying i wasn't invited but i got the invite [08:06:22] all right! [08:06:25] so we are going [08:06:26] Hahahaha. Right. [08:07:09] ori: let me know when you buy the ticket [08:07:22] we are going to have to buy some hipster glasses to fit in [08:08:26] heh [08:08:40] you're apparently not welcome to ask questions! [08:08:48] anyways, you were just about settling on perl [08:08:51] don't let me interrupt [08:09:00] that would be a fun outcome [08:09:15] settling on perl, i mean -- not letting me interrupt [08:10:41] please tell me we're not actually going to settle on perl ;p [08:10:49] Oh perl it is. [08:10:53] perl 6, even [08:11:05] Ironholds: you're to late. We'll settle on Perl. [08:11:14] * Ironholds whimpers [08:11:21] i hope you like parrot vm, Ironholds [08:11:22] I can barely, /barely/ read awk without going mad. [08:11:34] Well ... we got to love perl in wikistats, so we're expanding ont it's use. [08:11:46] the more you troll me the slower replacing wikistats will be :P [08:11:55] Hahaha :-D [08:12:09] If I come back in 3 weeks with this in my head, Toby will go "have you got the definition?" [08:12:18] "No" "well what the hell have you been doing?" "Worrying about Perl" [08:13:31] "That's not an excuse! Perl is beautiful! [30 minutes of Toby waxing lyrical on Perl, concluding in a decision to take your joke at face value, and then everyone has to write perl forever]" [08:14:06] Think about the alternatives ... writing the backup in ... say R :-P [08:15:07] hey, it'd be beautiful and have clean syntax and proper support for a C++ API and syntactic sugar around it [08:15:07] * qchris stops troll mode. [08:15:19] sure, it takes a week to run print(), but... [08:15:25] you know, every language has tradeoffs. [08:15:31] Hahaha. [09:04:36] (PS7) Nuria: 1st run of recurrent report is scheduled right away [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/138109 (https://bugzilla.wikimedia.org/66280) [09:33:18] qchris: had to do some changes to my last patch but i am back to the backup now, CR on last patchset is complete right? [09:33:32] no. [09:33:47] IIRC, I didn't review PS5. [09:34:24] We are talking about https://gerrit.wikimedia.org/r/#/c/139557/ [09:34:30] aren't we? [09:35:00] But since you're talking over, feel free to review/improve yourself upon milimetrics PS5. [09:36:11] Like guarding against symlinks, discussing rsync vs. cp more (or at least mare rsync act more robuist), ... [09:44:41] ok i will try to see what is going on, thnks [09:58:54] qchris: question about lab instances and time [09:58:56] ori, nuria: Do we know of anyone (except maintainers of EventLogging) using the {client,server}-server-events.log files (those under /a/eventlogging and the like) [09:59:14] nuria: Let's hear it. [09:59:19] yes the researchers [09:59:24] dario and aaron [09:59:44] they use the /a/eventlogging in 1002 [09:59:47] In addition to the log database they use the raw files? [10:00:01] Ok. Thanks. [10:00:13] sometimes they have yes, but on teh ocasional basis, we had an event were we repopulated from logs [10:00:20] and resercah looked at those then [10:00:23] *reserach [10:00:44] we had several weeks of data loss a while back [10:00:59] regarding times [10:01:07] what time do lab instances have? [10:01:24] 10am utc? [10:01:34] that is not gmt, right? [10:01:59] :-D [10:02:04] It's not the same, right. [10:02:21] But does the difference matter? [10:02:42] It matters cause wikimetrics is reporting utc times and it is making users very confuse i think [10:02:54] we should be in NY time, GMT or PST [10:03:01] ? [10:03:17] cause those *very confused [10:03:19] Do you understand the difference between gmt and utc? [10:03:48] For practical purposes, they should be close enough. [10:04:03] I thought it was GMT on a 24 hr schedule [10:04:15] but it is not [10:04:30] as now >date on labs instances prints [10:04:38] Wed Jun 18 10:01:11 UTC 2014 [10:06:00] Sorry. I do not understand the question. [10:06:12] Is it about the string representation of times on labs? [10:06:19] ah wait, it does NOT have daylight savings time [10:07:09] no, the question was about using a timezone that is just a reference, doesn't match any geographical zone [10:08:48] Like in the representation that the user sees or that the labs instances use internally? [10:09:08] The UI, printing utc times in the ui does not make much sense [10:10:54] I do not have much of an opinion here. [10:11:07] As long as the user sees the used timezone. [10:11:15] And the timezone is one that is generally understood. [10:11:29] And the user can adjust the timezone if thon wants to, [10:11:37] It's fine by me. [10:12:14] (CR) Nuria: Always display time zone (1 comment) [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/140222 (https://bugzilla.wikimedia.org/63923) (owner: Milimetric) [10:12:24] But IIRC, users currently cannot set the used timezone. [10:12:53] So I'd be transparent about which timezone gets used, and pick a timezone that's +0. [10:13:06] Right, we were fixing that thus my piint of UTC not being a timezone right now. [10:18:28] (CR) QChris: Always display time zone (1 comment) [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/140222 (https://bugzilla.wikimedia.org/63923) (owner: Milimetric) [10:57:23] (CR) Nuria: Always display time zone (1 comment) [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/140222 (https://bugzilla.wikimedia.org/63923) (owner: Milimetric) [11:03:58] qchris: I just downloaded the backup changes (plus a rebase) as it looks like some chnages were merge yesterday and I get: [11:03:59] Duplicate definition: Package[redis-server] is already defined in file /tmp/vagrant-puppet-1/modules-0/redis/manifests/init.pp at line 67; cannot redefine at /tmp/vagrant-puppet-1/modules-0/wikimetrics/manifests/queue.pp:18 on node mediawiki-vagrant.home [11:04:51] Redis setup was changed yesterday by milimetric and ottomata. [11:05:05] That looks like it could be related. [11:09:05] (CR) QChris: Always display time zone (1 comment) [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/140222 (https://bugzilla.wikimedia.org/63923) (owner: Milimetric) [11:18:04] qchris: ya, it is, removing their change still gives out this error: Puppet::Parser::AST::Resource failed with error ArgumentError: Invalid resource type apache::mod at /tmp/vagrant-puppet-1/modules-0/wikimetrics/manifests/web/apache.pp:21 on node mediawiki-vagrant.home [11:18:29] so something else changed on apache's config [11:18:43] Well ... apache config is getting remodelled. [11:19:02] That also happened yesterday, along with a discussion around wikimetrics. [11:19:11] You'll find it in yesterday's logs. [11:19:22] the dev environment is broken [11:24:37] well, no actually they all are [11:25:08] if you upgrade puppet production branch wikimetrics does not start due to the apache error [11:26:13] (CR) Gilles: [C: -1] "This isn't enough, since the media viewer changeset bumps the schema version to 8922098. You need to UNION with the new MediaViewer_892209" [analytics/multimedia] - https://gerrit.wikimedia.org/r/140263 (owner: Gergő Tisza) [11:27:11] (CR) Gilles: "Feel free to +2 this one once the SQL changeset is merged." [analytics/multimedia/config] - https://gerrit.wikimedia.org/r/140264 (owner: Gergő Tisza) [11:28:12] nuria: metrics.wmflabs.org seems up [11:29:04] (Tried creating cohort, running report, looking at the json result) [11:29:22] sorry, yes it is up, we just will not be able to deploy chnages to it as the last changes in puppet/production branch break the env [11:29:28] Ok. [11:30:43] IIRC milimetric and ottomata decided that redis will not be configured through puppet, but by hand. [11:30:58] But I am sure you found those commits/discussions already. [11:31:19] If you need help fixing, let me know. Brb. [11:31:30] the redis commit breaks vagrant [11:31:37] as far as i can see [11:32:00] but i think milimetric test it there so will wait to talk to him i guess [11:37:00] Back. [11:38:00] Regarding the redis commit and vagrant ... You did provide a redis configuration by hand, right? [12:04:41] nuria: I am having issues connecting to stat1002 and stat1003 ... can you log in to them? [12:08:43] nuria: Issue seems gone... whatever it was... Sorry for the noise. [12:16:49] np [12:17:15] that was happening to me couple days ago remember? [12:18:54] qchris : question [12:19:07] on the backup scripts.... [12:19:13] do we want 755 permits? [12:19:16] https://gerrit.wikimedia.org/r/#/c/139557/6/manifests/backup.pp [12:49:21] nuria: That depends a bit on what user/group we settle on. [12:49:47] Group has lost write ownership. That was the important part from PS4. [12:50:02] s/PS4/PS3/ [12:50:49] If we settle with allowing the group to execute, we should make sure that the script [12:51:07] works when run by a group-member that is not the owner. [12:52:28] I'd be great if we could solve "require root" issue and use proper priviledges. [12:53:27] s/I'd/It'd/ [12:53:43] * qchris applies for typing/spelling lessons :-( [12:53:59] * YuviPanda gives qchris some privi ledges [12:54:13] qchris: ok, let me read all comments [12:54:35] to all patchsets [12:54:48] * qchris offers his other cheek to YuviPanda [12:58:04] qchris: :) don't jump off the priviledges! [12:58:23] :-D [13:00:16] 1676, Isaak Newton, Philosophical Transactions of the Royal Society of London[1], page 561: if he eſteem it ſo extraordinary , he may have the priviledg of making it ſo viſible to all , that it will be a ſhame to deny it. [13:00:49] Nemo_bis: missing an 'e' [13:00:54] Does Wiktionary really need to use ſ? *shrug* [13:00:58] YuviPanda: tell Isaak [13:15:08] qchris: read all comments [13:15:26] where does destination [13:15:31] get set? [13:15:44] You mean in https://gerrit.wikimedia.org/r/#/c/139557/6/manifests/backup.pp ? [13:15:46] now we have: /manifests/backup.pp: $destination = undef, [13:15:52] yes [13:16:19] That is just the definition of the class. [13:16:27] Instantiation happens in a different change. [13:16:32] https://gerrit.wikimedia.org/r/#/c/139558/2/manifests/role/wikimetrics.pp [13:16:45] ^ line 202 sets $destination [13:20:38] I see, then from having read all comments the issues still open are [13:20:45] 1. whether we need root [13:21:27] 2. redis backup file name hardcoded (acording to dan not easy to fix) [13:21:55] 3. rsync parameters [13:22:04] 4. symlinks on rsync [13:23:41] symlinks do not only affect us for rsync, but for all files. [13:23:50] 5. i also think error messages on script execution could use some help, printing usage is not enough [13:24:03] And those are only the issues we identified up to now. There might be more to come. [13:25:21] Due to comments on line 48 at https://gerrit.wikimedia.org/r/#/c/139557/3/files/backup_daily [13:25:33] I am not sure if milimetric agrees to you're item #5. [13:27:22] well.. i am afraid I am one of those that needs 'separate errors' , [13:27:28] i just executed it like: ./backup_daily_exec -i ./input -o ./output -k 3 [13:28:04] and would have been nice to know what was not working [13:28:22] but , leaving that aside, [13:28:51] what is the issue with symlinks (besides them needing specific args in rsync) [13:31:13] ^qchris [13:31:46] Backing up symlinks as symlinks might do the wrong thing, if for example files get big, hence get replaced by symlinks to files on different mounts. [13:32:03] The we have the symlink to the file we are interested in in the backup. [13:32:11] And not the real file we are interested in. [13:32:44] On the other hand, we need to make sure that symlinks are treated correctly in the public-files directory, [13:33:02] if we decide to link the cryptic public-recurrent-report directories to more descriptive names. [13:33:25] Solutions to those problems are probably easy [13:33:32] But we just need to think it through. [13:34:37] I do not see us doing symlinks on teh filesytem to link the cryptic public-recurrent-report directories to more descriptive names. [13:35:11] The other day, we considered it :-) [13:35:26] But I am not challenging it. If we do not, that's fine. [13:35:38] We just need to make sure we thought about it. [13:35:39] We will not , that is much better than at the request level [13:35:54] that is what routing is for [13:36:11] we do not need the filesystem for that [13:36:24] so ruling that out we are just left [13:36:28] Well ... it was just an example. I just want to make sure, we thought about symlikns. [13:37:12] with "undoing" symlinks so as not backup symlinks that point to elsewhere on teh machine, correct? [13:37:58] Right. But we cannot do it unconditionally due to possible issue as mentioned above. ... or we do it unconditionally for now and [13:38:13] add a check that alerts us if current assumptions are no longer valid. [13:38:32] Or some other $WICKED_COOL_THING. [13:38:43] Right, to keep things simple that is what we will do for now. [13:38:47] hahaha [13:38:59] loved that +>$WICKED_COOL_THING. [13:42:49] so , i voting for rsync -L [13:47:08] agh, ori, ha, stat1001 was actually using 000-default! [13:47:15] ori: you still around? [13:47:23] ottomata: Ha. Just wanted to ask the same thing. [13:47:39] fixing... [13:47:42] Thanks. [13:47:59] Corresponding bug: https://bugzilla.wikimedia.org/show_bug.cgi?id=66781 [14:36:55] (CR) Milimetric: Always display time zone (1 comment) [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/140222 (https://bugzilla.wikimedia.org/63923) (owner: Milimetric) [14:37:19] (CR) QChris: [C: 2] Always display time zone [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/140222 (https://bugzilla.wikimedia.org/63923) (owner: Milimetric) [14:37:28] (Merged) jenkins-bot: Always display time zone [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/140222 (https://bugzilla.wikimedia.org/63923) (owner: Milimetric) [14:47:30] (CR) Milimetric: [C: 2] "woo hoo!" [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/138109 (https://bugzilla.wikimedia.org/66280) (owner: Nuria) [15:03:18] qchris: how do I add a bugzilla link to a commit message [15:03:27] BUG 63664 [15:03:27] ? [15:04:12] Bug: $BUG_NR [15:04:15] ottomata: [15:04:34] Put that in the same paragraph as the "Change-Id:" [15:04:53] (or if there is no "Change-Id:" yet, put it on the last line of the commit message) [15:04:55] k [15:05:13] https://www.mediawiki.org/wiki/Gerrit/Commit_message_guidelines#Example [15:05:20] danke [15:18:22] ok milimetric, i think staging looks good [15:18:25] can you double check? [15:18:32] if so, i'll update pupppet on prod and run it [15:20:18] thanks ottomata for fixing vagrant [15:21:35] ottomata, question if you have time [15:24:22] ja what's up? [15:30:17] sooo [15:30:30] we have this backup chnageset [15:30:37] *changeset [15:30:47] And we wnat to backup redis db [15:31:06] see line #22 of https://gerrit.wikimedia.org/r/#/c/139557/7/manifests/backup.pp [15:31:17] ^ottomata [15:31:42] ja, k [15:32:04] we would like not have to hardcode the name of the file [15:32:17] a/redis/wikimetrics1-6379.rdb [15:33:13] but millimetric was saying it is hard to get that per host, is there a way that file can abide to a convention and be [15:33:35] a/redis/wikimetrics-backup.rdb so it has teh same name in every host [15:33:53] that or retrieve the file name per host [15:36:42] yeah that'd be nice [15:39:56] ^ottomata [15:42:30] qchris, see line 66 of https://gerrit.wikimedia.org/r/#/c/139557/6/manifests/backup.pp [15:42:50] the crontab needs to be set up daily right? [15:43:52] nuria: I think so. Daily for the daily script looks right. [15:45:40] nuria, i don't know how redis generates the db names [15:45:46] maybe there is some value in the config file? [15:45:55] now that the config file is puppetized specifically for wikimetrics [15:45:57] it should be easier to change [15:46:09] oh [15:46:09] For the rdb file? It is in the config template. [15:46:13] is it hostname-port? [15:46:15] or something? [15:46:17] Yes. [15:46:34] could we chnage that to be backup-port [15:46:42] backup-$port [15:46:43] oh, i hardcoded that in, and it shouldnt' be [15:46:47] well, yeah, you could [15:47:01] nuria, see templates/wikimetrics/redis.conf.erb in ops/puppet [15:47:06] line 126 [15:47:10] change that to whatever you want :) [15:47:15] ok, will change [15:48:00] and ... one last question... [15:48:08] milimetric: does staging redis and wikimetrics stuff look ok? [15:49:44] how do you test crontabs on puppet? [15:49:51] do stuff goes to /etc/cron.daily/ [15:50:57] ? [15:51:41] no, to the user's crontab [15:51:43] so [15:51:46] i usually check [15:51:51] sudo -u crontab -l [15:54:08] evn if user is root? [15:55:36] if user is root [15:55:40] sudo crontab -l [15:55:41] will do it [15:55:46] but sure [15:55:50] sudo -u root works too [15:58:52] ottomata: myquestion was rather : can we use cronbtabs as root? [15:59:10] can we 'set' crontabs for the root user [15:59:16] ja sure [16:17:40] ottomata: i won't know for 100% sure for a day, but the settings that I want to be there are there [16:17:55] thanks again [16:18:10] ok cool, safe to run in prod then? [16:18:25] actually, i'll wait til nuria changes that rdb fiel setting? [16:18:57] i was going to do it with the backup stuff but i can do it right now [16:19:32] yeah, make it a separate commit I think? will it affect anythign in prod? [16:19:41] e.g., do we need to move the current file to the new name? [16:19:51] no, our db file in prod is broken (as there is no data) [16:20:14] we should do a shutdown though and bring up gain and rremove old file [16:22:24] k [16:31:27] i just changed teh name but i guess to be trully good it should be a variable on wikimetrics.pp role, is that so? [16:36:50] naw, in this case it doesn't really matter [16:36:56] since we are passing our hardcoded redis template in [16:37:02] this template is 100% for wikimetrics [16:37:10] and, the redis module itself is the one that renders the template [16:37:22] so you'd have to to some trickiness to make any wikimetrics parameters in local scope in the redis class [16:37:34] or have the template itself know how to look them up with global resolution [16:39:45] i see [16:43:41] I would need to define it in wikimetrics.pp and have redis template and wikimetrics look that variable up in global scope [16:46:31] yeah, i won't worry abou tit [16:46:36] since we have our own hardcoded template now [16:46:37] for wikimetrics [17:40:56] DarTar: Schema:Edit! I promise it can be as short as 10 min... [17:43:52] awight: yes, are you around at 3.30-ish? the rest of the day is pretty booked [17:44:23] milimetric: stress-testing wikimetrics with large cohorts, it’s doing well so far [17:45:04] nuria should know that too, because we were going to deploy a change and stop the system in the process [17:45:13] nuria - probably just wait to see the queue stops processing then [17:45:13] ha [17:45:48] I’m running this for lzia, who is presenting in an hour from now, any chance you guys could wait until then? [17:45:53] definitely [17:46:03] it should take less [17:46:07] thanks [17:46:12] ottomata: just in case you were thinking of deploying wikimetrics, let's hold off for a couple of hours [17:46:19] looks like research needs it [17:46:54] k [17:52:18] DarTar: sure, I'll even calendar it [17:53:56] ottomata: sonofagun :) [17:54:05] damn redis still deleting data [17:54:23] it's good we can configure it now, but which config to run is a mystery [17:54:29] did y'all restart it when you deployed the changes? [17:54:33] not wikimetrics, but redis-server [17:55:29] ? [17:55:40] i haven't run puppet on prod [17:55:42] only on staging [17:55:48] and no, I did not restart redis [17:59:43] milimetric: out of curiosity, any reason while ID validation is done in batches of 1K users? [17:59:53] performance [18:00:06] kk [18:01:33] I ran a 110K user aggregate report for bytes added and it was pretty fast [18:05:53] Hey all, how's it going? Anyone know about new errors in wikimetrics tests? I'm getting two errors in tests on master as of eb3ccb8854ded032fd195b35e06c48e51762d376 "Add Newly Registered Users metric" [18:10:16] (PS4) AndyRussG: Improve server-side cohort upload form validation [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/138151 [18:18:43] (CR) AndyRussG: "Just rebased, fixing conflict..." [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/138151 (owner: AndyRussG) [18:52:28] hey AndyRussG [18:52:37] yes, the two errors are known problems, there's a bug filed [18:52:52] it's no big deal - the alembic migrations fell out of sync with one of the tests [18:59:24] milimetric: Ah OK thanks! Was just wondering... [18:59:34] I guess I should have checked the bug reports [18:59:41] no, it's ok to ask, no problem [18:59:50] we don't expect you to keep up with the nutty bug stream :) [19:04:36] Ah OK [19:18:39] Hmmm nutty bug stream... like something that flows through a tube in a cricket-flavored-peanut-butter factory? [20:01:40] (PS13) AndyRussG: WIP Create a cohort from campaign participations [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/126927 (owner: Awight) [20:02:09] (PS3) Gergő Tisza: Add above-fold filepage link to stats fields [analytics/multimedia] - https://gerrit.wikimedia.org/r/140263 [20:03:26] (CR) AndyRussG: "Rebased, fixing conflicts." [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/126927 (owner: Awight) [20:18:41] (PS2) Milimetric: Fix Aggregation of empty metric results [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/138031 (https://bugzilla.wikimedia.org/66439) (owner: Nuria)