[00:20:20] I'm trying to curl this service running on krypton on port 8000 from stat1002 - (port was opened up here https://gerrit.wikimedia.org/r/#/c/249433/2/manifests/role/analytics/burrow.pp) The connection times out though - does anyone know what I'm missing? [00:20:26] uhhh [00:20:32] meant to ask on ops [01:23:12] Analytics, Discovery, Reading-Infrastructure-Team: Determine proper encoding for structured log data sent to Kafka by MediaWiki - https://phabricator.wikimedia.org/T114733#1942809 (bd808) I talked with @ottomata briefly a couple of weeks ago (in the lobby of Club Quarters in SF) and he mentioned that f... [01:35:24] Analytics, Analytics-Cluster, EventBus, Services, operations: Investigate proper set up for using Kafka MirrorMaker with new main Kafka clusters. - https://phabricator.wikimedia.org/T123954#1942833 (GWicke) Cross-DC consumption without replication would mean that events from an unavailable DC w... [01:51:01] Analytics, Analytics-Cluster, EventBus, Services, operations: Investigate proper set up for using Kafka MirrorMaker with new main Kafka clusters. - https://phabricator.wikimedia.org/T123954#1942844 (mobrovac) The point for me here are not names, but rather the ability of consumers in different... [08:02:03] Analytics-Kanban: Add luca to analytics e-mail alias [1] - https://phabricator.wikimedia.org/T123941#1943257 (elukey) [08:02:55] Analytics-Kanban: Add luca to analytics e-mail alias [1] - https://phabricator.wikimedia.org/T123941#1943261 (elukey) p:Triage>Normal [08:44:44] hello! [09:10:50] Analytics-Backlog, operations: Create email alias that will send emails to all Analytics Engineers - https://phabricator.wikimedia.org/T121180#1943351 (akosiaris) Added Luca Toscano (new analytics hire) as well [09:19:56] Analytics-Kanban: Add luca to analytics e-mail alias [1] - https://phabricator.wikimedia.org/T123941#1943352 (elukey) Solved in https://phabricator.wikimedia.org/T121180. I don't have access to iron.wikimedia.org and therefore I can't makes changes to palladium's private repo (as depicted in https://wikitech.... [09:20:11] Analytics-Kanban: Add luca to analytics e-mail alias [1] - https://phabricator.wikimedia.org/T123941#1943353 (elukey) Open>Resolved [09:36:54] Good morning elukey :) [09:48:11] joal: hello! [10:02:09] || 2016-01-17T09/1H || . | . | . | . | 0.0% || _ | _ | _ | _ | _ || [10:02:14] all clear :P [10:02:28] (finally I am receiving the alers) [10:02:42] Great :) [10:21:52] Kafka brokers seems to get along in the last two days https://grafana.wikimedia.org/dashboard/db/kafka?panelId=16&fullscreen [10:22:09] *seem [10:22:31] elukey: they usually work fairly well :) [10:22:48] I know but I wanted to investigate why they were lagging :P [10:22:59] right :) [10:23:00] I'll ready all the docs and comments from Andrew [10:23:18] yeah, he's the one knowing better [10:23:36] From what I understood, it's only affecting topics that have one partition IIRC [10:32:08] do we usually use one partition? Or is there a guideline about it? Just realized it after sending the email to analytics-internal [10:32:11] for event bug [10:32:13] *bus [10:34:05] elukey: I think your comments to ottomata are good :) [10:34:27] elukey: about partitionning in kafka, it depends on the epxected data size [10:34:55] elukey: for the moment mostly ottomata and a-team decide based on expected data size [10:36:59] ah snap I mixed the numbers in my reply, it was number 2) the unclear one :( [10:37:50] a-team: sorry I need to pay attention before sending emails, especially when under-caffeinated and jet-lagged. Please be patient and tell me "it is gonna be all right". [10:38:01] :D [10:38:04] :D [11:14:33] Analytics, Discovery, Reading-Infrastructure-Team: Determine proper encoding for structured log data sent to Kafka by MediaWiki - https://phabricator.wikimedia.org/T114733#1943492 (dcausse) The pipeline is working fine for us with avro binary now. Erik wrote a documentation about it : https://wikitech.... [11:50:26] (PS2) Joal: Use webrequest_source text for AppSessionMetrics, mobile is merging with text [analytics/refinery/source] - https://gerrit.wikimedia.org/r/264868 (https://phabricator.wikimedia.org/T122651) (owner: Ottomata) [11:53:51] Quarry: Make the table sortable - https://phabricator.wikimedia.org/T71265#1943595 (He7d3r) [11:58:25] (PS3) Joal: Use webrequest_source text for AppSessionMetrics, mobile is merging with text [analytics/refinery/source] - https://gerrit.wikimedia.org/r/264868 (https://phabricator.wikimedia.org/T122651) (owner: Ottomata) [12:12:57] (CR) Joal: [C: 2] "Self merging to prepare deploy with @ottomata later." [analytics/refinery/source] - https://gerrit.wikimedia.org/r/264868 (https://phabricator.wikimedia.org/T122651) (owner: Ottomata) [12:20:50] (PS4) Mforns: Add split-by-os argument to AppSessionMetrics job [analytics/refinery/source] - https://gerrit.wikimedia.org/r/264297 (https://phabricator.wikimedia.org/T117615) [12:21:07] (CR) jenkins-bot: [V: -1] Add split-by-os argument to AppSessionMetrics job [analytics/refinery/source] - https://gerrit.wikimedia.org/r/264297 (https://phabricator.wikimedia.org/T117615) (owner: Mforns) [12:29:30] (PS5) Mforns: Add split-by-os argument to AppSessionMetrics job [analytics/refinery/source] - https://gerrit.wikimedia.org/r/264297 (https://phabricator.wikimedia.org/T117615) [12:31:44] (CR) jenkins-bot: [V: -1] Add split-by-os argument to AppSessionMetrics job [analytics/refinery/source] - https://gerrit.wikimedia.org/r/264297 (https://phabricator.wikimedia.org/T117615) (owner: Mforns) [12:35:23] (PS1) Joal: Clean repo from mobile partition leftovers [analytics/refinery/source] - https://gerrit.wikimedia.org/r/264940 (https://phabricator.wikimedia.org/T122651) [12:35:31] mforns: Hi ! [12:35:37] joal, hi! [12:35:45] everythink OK? [12:35:51] *thing [12:35:54] mforns: Would you mind having a look at the previous change I submitted ? [12:36:06] sure [12:36:16] mforns: I love "everythink ok", put me in good mood :) [12:36:52] :] [12:38:30] (CR) jenkins-bot: [V: -1] Clean repo from mobile partition leftovers [analytics/refinery/source] - https://gerrit.wikimedia.org/r/264940 (https://phabricator.wikimedia.org/T122651) (owner: Joal) [12:39:22] joal, I think jenkins didn't like the change... [12:39:37] looks like so mforns ! [12:39:53] mine also... [12:41:21] (PS4) Mforns: Divide app session metrics job into global and split [analytics/refinery] - https://gerrit.wikimedia.org/r/264292 (https://phabricator.wikimedia.org/T117615) [12:41:55] (CR) Mforns: [C: -1] "Still WIP" [analytics/refinery] - https://gerrit.wikimedia.org/r/264292 (https://phabricator.wikimedia.org/T117615) (owner: Mforns) [12:42:21] mforns: I actuqally forgot one thing (my bad really) [12:42:35] Will submit again in a minute [12:42:37] ok [12:43:57] And will offer a beer at next off-site (the one who breaks the build ...) [12:48:55] xD [12:49:24] (PS2) Joal: Clean repo from mobile partition leftovers [analytics/refinery/source] - https://gerrit.wikimedia.org/r/264940 (https://phabricator.wikimedia.org/T122651) [12:54:48] (PS6) Mforns: Add split-by-os argument to AppSessionMetrics job [analytics/refinery/source] - https://gerrit.wikimedia.org/r/264297 (https://phabricator.wikimedia.org/T117615) [12:58:14] (CR) Mforns: [C: 2 V: 2] "LGTM!" [analytics/refinery/source] - https://gerrit.wikimedia.org/r/264940 (https://phabricator.wikimedia.org/T122651) (owner: Joal) [12:58:22] joal: I forgot to tell you! I found a good sushi bar nearby the office called Akiko Sushi. There are 10 chairs available but it is worth it :) [12:58:34] Haahahaha :) [12:58:44] elukey: next year, if it's still on :) [12:59:10] (PS7) Mforns: Add split-by-os argument to AppSessionMetrics job [analytics/refinery/source] - https://gerrit.wikimedia.org/r/264297 (https://phabricator.wikimedia.org/T117615) [13:04:49] (PS1) Joal: Update changelog for v0.0.25 deployment [analytics/refinery/source] - https://gerrit.wikimedia.org/r/264941 [13:04:55] Thanks mforns for the review [13:05:01] mforns: if you don't mind --^ [13:09:06] (CR) Mforns: [C: 2 V: 2] "LGTM" [analytics/refinery/source] - https://gerrit.wikimedia.org/r/264941 (owner: Joal) [13:09:57] Thanks again mforns :) [13:10:04] np :] [13:12:14] (CR) Mforns: [C: -1] "Still WIP" [analytics/refinery/source] - https://gerrit.wikimedia.org/r/264297 (https://phabricator.wikimedia.org/T117615) (owner: Mforns) [13:47:57] (PS2) Joal: Remove mobile webrequest_source merging it in text [analytics/refinery] - https://gerrit.wikimedia.org/r/264870 (https://phabricator.wikimedia.org/T122651) (owner: Ottomata) [14:00:19] (PS1) Joal: Bump jars version to 0.0.25 [analytics/refinery] - https://gerrit.wikimedia.org/r/264948 [14:03:05] Analytics-Tech-community-metrics, Phabricator, User-greg: Closed tickets in Bugzilla migrated without closing event? - https://phabricator.wikimedia.org/T107254#1943852 (Aklapper) @greg: Any comments on T107254#1795257 ? [14:08:37] (PS2) Joal: Bump jars version to 0.0.25 [analytics/refinery] - https://gerrit.wikimedia.org/r/264948 (https://phabricator.wikimedia.org/T122651) [14:09:03] (PS1) Joal: Modify oozie job to use jars v0.0.25 [analytics/refinery] - https://gerrit.wikimedia.org/r/264949 (https://phabricator.wikimedia.org/T122651) [14:11:25] (CR) Joal: [V: 2] Correct pageview oozie job [analytics/refinery] - https://gerrit.wikimedia.org/r/259661 (owner: Joal) [14:12:34] (PS3) Joal: Remove mobile webrequest_source merging it in text [analytics/refinery] - https://gerrit.wikimedia.org/r/264870 (https://phabricator.wikimedia.org/T122651) (owner: Ottomata) [14:13:11] (PS3) Joal: Bump jars version to 0.0.25 [analytics/refinery] - https://gerrit.wikimedia.org/r/264948 (https://phabricator.wikimedia.org/T122651) [14:13:37] (PS2) Joal: Modify oozie job to use jars v0.0.25 [analytics/refinery] - https://gerrit.wikimedia.org/r/264949 (https://phabricator.wikimedia.org/T122651) [14:17:36] Analytics, Analytics-Cluster, EventBus, Services, operations: Investigate proper set up for using Kafka MirrorMaker with new main Kafka clusters. - https://phabricator.wikimedia.org/T123954#1943905 (GWicke) > I guess in a first phase we could manually instruct them to do so, but automatising co... [14:34:55] Analytics, Analytics-Cluster, Patch-For-Review: Single Kafka partition replica periodically lags - https://phabricator.wikimedia.org/T121407#1943942 (elukey) From https://issues.apache.org/jira/browse/KAFKA-2477 it seems that the original issue is a series of deletions for a partition terminated by a... [14:36:51] Analytics, Discovery, Reading-Infrastructure-Team: Determine proper encoding for structured log data sent to Kafka by MediaWiki - https://phabricator.wikimedia.org/T114733#1943952 (Ottomata) Whoa, I had not seen that page. Nice job Erik, that is really great! [14:37:56] * elukey fights with grafana to compose graphs with multiple metrics [14:45:44] Analytics, Analytics-Cluster, Patch-For-Review: Single Kafka partition replica periodically lags - https://phabricator.wikimedia.org/T121407#1943969 (elukey) I can see also spikes in "under replicated partitions" at the same time of the lags (kinda expected). It seems that this issue is indeed caused... [14:52:31] Analytics, Analytics-Cluster, Patch-For-Review: Single Kafka partition replica periodically lags - https://phabricator.wikimedia.org/T121407#1943981 (elukey) Link to the explanation in the task: https://issues.apache.org/jira/browse/KAFKA-2477?focusedCommentId=14734928&page=com.atlassian.jira.plugin.... [15:04:34] (CR) Ottomata: [C: 2 V: 2] Bump jars version to 0.0.25 [analytics/refinery] - https://gerrit.wikimedia.org/r/264948 (https://phabricator.wikimedia.org/T122651) (owner: Joal) [15:06:34] (CR) Ottomata: [C: 2 V: 2] Modify oozie job to use jars v0.0.25 [analytics/refinery] - https://gerrit.wikimedia.org/r/264949 (https://phabricator.wikimedia.org/T122651) (owner: Joal) [15:07:06] oh ha, just realized those depend on mine which is not merged [15:07:38] (CR) Ottomata: "Yeah, Madhu, I didn't bother. The tests are contained and don't actually reference real data." [analytics/refinery] - https://gerrit.wikimedia.org/r/264870 (https://phabricator.wikimedia.org/T122651) (owner: Ottomata) [15:07:40] hehe ottomata :) [15:08:15] ottomata: I have pushed changes on top of yours (including madhu's comment) [15:08:27] joal: how's that change look to you? [15:08:28] cool ja [15:08:35] what'd you change? [15:08:38] ottomata: I think we're ready to go [15:08:58] k let's check in with bblack [15:09:23] ottomata: I have carefully reviewed your patch, double checked the repo again and added the few missing changes --> hoppefully it'll go smoothely [15:28:02] RECOVERY - Overall insertion rate from MySQL consumer on graphite1001 is OK: OK: Less than 20.00% under the threshold [100.0] [15:28:40] joal: no word from bblack yet, i guess it is still early. [15:28:50] np ottomata ) [15:28:51] Analytics-Kanban, DBA, Patch-For-Review: EL replication having issues since at least January 11th - https://phabricator.wikimedia.org/T123634#1944055 (jcrespo) I think all issues are gone, except on MobileAppUploadAttempts_5257716, which may be a false positive or something else that makes the check co... [15:29:25] ottomata: shall go and merge everything, preparing for deploy ? [15:30:36] yeah let's [15:30:43] he jsut pinged me, he's starting in 1.5 hish [15:31:22] k ottomata [15:31:36] Analytics-Kanban, operations, HTTPS, Patch-For-Review: EventLogging sees too few distinct client IPs {oryx} [8 pts] - https://phabricator.wikimedia.org/T119144#1944056 (Ottomata) @ironholds or @tbayer, can you confirm that `clientIP`s make more sense now? [15:32:12] ottomata: I have meetings in 1h, for 2h (end of stqaff meetings) [15:33:59] Analytics, Analytics-Cluster, EventBus, Services, operations: Investigate proper set up for using Kafka MirrorMaker with new main Kafka clusters. - https://phabricator.wikimedia.org/T123954#1944062 (Ottomata) Ok, so it sounds like the master-master with topics named after DCs is needed then, yes? [15:34:04] its ok [15:34:07] we don't need to do anything then [15:34:09] he's just startgin then [15:34:21] Analytics-Kanban, DBA, Patch-For-Review: EL replication having issues since at least January 11th - https://phabricator.wikimedia.org/T123634#1944063 (jcrespo) Open>Resolved a:jcrespo There is definitely something wrong with that table, but it is not related to replication: ``` root@iron:~$ m... [15:35:13] OK so, milimetric [15:35:44] I seem to have a built dashiki interface, but no data. Probably because I don't have things in the correct magical places. I need to be on stat1002 for that, or will 1003 work? [15:36:28] MarkTraceur: which config are you using? [15:36:47] metrics-by-project/MultimediaHealth [15:37:02] It's probably just not complete actually [15:38:00] https://meta.wikimedia.org/wiki/Config:MultimediaHealth [15:39:03] Analytics, Analytics-Cluster, EventBus, Services, operations: Investigate proper set up for using Kafka MirrorMaker with new main Kafka clusters. - https://phabricator.wikimedia.org/T123954#1944072 (GWicke) @ottomata, I think in conventional replication terminology the mirrormaker stuff is all... [15:39:03] cool, (reading config) [15:39:12] It's a short read [15:40:53] MarkTraceur: ok, so you had set up https://meta.wikimedia.org/wiki/Dashiki:CategorizedMetrics before or just now? [15:41:01] that has definitions for your metrics [15:41:20] That was before [15:41:29] and from that, it looks like it's pointing to the default site which is datasets.wikimedia.org, hang on, checking :) [15:41:35] Oh, this is why there were directories [15:41:50] aha, MarkTraceur so this: http://datasets.wikimedia.org/limn-public-data/metrics/ [15:41:57] Yeah [15:42:03] that's where it expects to find what you defined in the Dashiki:CategorizedMetrics [15:42:22] I have it there, but it's all in all/multimedia-health [15:42:26] in your case, it'll look for multimedia-health/uploads, multimedia-health/uploaders, etc. [15:42:43] So I think I need to tweak the generation script to put stuff into the appropriate directory trees per-project [15:42:56] all? [15:43:03] http://datasets.wikimedia.org/public-datasets/all/multimedia-health/ [15:43:24] oh I see, no, this thing right now looks in /limn-public-data/metrics/ [15:43:41] and when we finally rename that I'll change that default in dashiki [15:44:13] MarkTraceur: but the way you have them in here is good: http://datasets.wikimedia.org/public-datasets/all/multimedia-health/uploaders/ [15:44:24] OK, perfect [15:44:35] so just moving that multimedia-health directory to /limn-public-data/metrics/ should do it [15:44:40] I'm just fixing the files and then the script [15:44:45] Oh, OK [15:44:46] MarkTraceur: for now if you just want to test you can just change that in dashiki [15:45:21] https://github.com/wikimedia/analytics-dashiki/blob/14044aec79e49fe8cf797817f6a0854bcae15b01/src/app/config.js#L40 [15:45:26] MarkTraceur: ^ [15:45:54] Naw, I'll wait the fifteen minutes or so, it'll be fine [15:48:04] OK, everything's moved around [15:48:16] Script should be fixed too [15:48:28] I'm thinking I should probably time-limit it though [15:50:01] OK, more to come after our standup [15:52:08] Analytics-Tech-community-metrics, Phabricator, User-greg: Closed tickets in Bugzilla migrated without closing event? - https://phabricator.wikimedia.org/T107254#1944107 (greg) >>! In T107254#1795257, @Aklapper wrote: > I think chasemp and I worked out the technical side of things in the last comments,... [15:52:58] Analytics, Analytics-Cluster, EventBus, Services, operations: Investigate proper set up for using Kafka MirrorMaker with new main Kafka clusters. - https://phabricator.wikimedia.org/T123954#1944108 (Ottomata) Indeed you are right, it is much more like master-slave since the topics are distinct.... [15:58:56] Analytics-Kanban, DBA, Patch-For-Review: Pending maintenance on the eventlogging databases (db1046, db1047, dbstore1002, other dbstores) - https://phabricator.wikimedia.org/T120187#1944127 (Nuria) @jcrespo: let us know if wednesday is a good day to do this scheduled maintenance and we will proceed to a... [16:11:13] Analytics-Kanban, DBA: Foundation-only Geowiki stopped updating - https://phabricator.wikimedia.org/T106229#1944177 (Milimetric) a:Milimetric>jcrespo [16:23:30] Analytics-Kanban, DBA, Patch-For-Review: Pending maintenance on the eventlogging databases (db1046, db1047, dbstore1002, other dbstores) - https://phabricator.wikimedia.org/T120187#1944235 (jcrespo) Let's announce it for Thursday (if you are ok with it), my backlog is larger than I thought :-) Do you... [16:30:25] OK milimetric, next up: I have one metric (uploads) but two others (uploaders, new-uploaders) won't show up [16:30:38] https://meta.wikimedia.org/wiki/Dashiki:MultimediaHealthUploads doesn't exist, so the other two not existing isn't an issue [16:34:32] (CR) Nuria: Clean repo from mobile partition leftovers (1 comment) [analytics/refinery/source] - https://gerrit.wikimedia.org/r/264940 (https://phabricator.wikimedia.org/T122651) (owner: Joal) [16:38:16] sorry, had my speakers off :P [16:38:26] milimetric: No problem [16:38:56] (reading the categorized metrics config) [16:39:30] the fact that https://meta.wikimedia.org/wiki/Dashiki:MultimediaHealthUploads doesn't exist shouldn't stop data from showing up, that's just for annotations [16:39:36] Right [16:40:14] MarkTraceur: you selected the Uploaders metric from the "Add Metrics" drop down thing, right? (no offense intended, just making sure that button's as obvious as I think) [16:40:32] Analytics-Kanban, DBA: Foundation-only Geowiki stopped updating - https://phabricator.wikimedia.org/T106229#1944281 (jcrespo) a:jcrespo>None I've set s1-analytics-slave.eqiad.wmnet as read-write. It was set as read only when maintenance was performed there 29.2 days ago. I have no problem on doing... [16:40:36] milimetric: I opened the drop down, neither Uploaders nor New Uploaders is present there [16:40:43] oh! I see... [16:40:45] I could have been more specific with my bug description [16:41:15] MarkTraceur: oh, it's simple, the names have to match: https://meta.wikimedia.org/wiki/Config:MultimediaHealth [16:41:26] Oops. [16:41:33] so you named it Uploaders on the categorized metrics config, and not the same there in your dashboard config [16:41:45] Analytics-Kanban: Foundation-only Geowiki stopped updating - https://phabricator.wikimedia.org/T106229#1944286 (jcrespo) [16:41:48] grr, sorry, I will fix all this stuff once Dashiki gets more mainstream [16:42:57] OK, fixed [16:43:02] Analytics-Kanban: Foundation-only Geowiki stopped updating - https://phabricator.wikimedia.org/T106229#1462437 (jcrespo) Not closing because probably you need to perform more tasks to fully fix this issue(?). [16:43:11] Working now except for the commons new uploaders because that crashed when I tried to run it [16:43:21] I'll try it again and see what happens [16:45:23] milimetric: I think last time we talked you showed me a script that would automatically run the script for each month, instead of recalculating all of the data each time, do you remember what that might have been? [16:46:15] run the sql query* [16:47:17] Analytics-Kanban: Foundation-only Geowiki stopped updating - https://phabricator.wikimedia.org/T106229#1944309 (Milimetric) @jcrespo, thanks, the scripts should catch up by themselves, but I'll close this when I confirm. I'll comment inline below: >>! In T106229#1944281, @jcrespo wrote: > I've set s1-anal... [16:48:14] MarkTraceur: yes, that's reportupdater, a python utility developed by mforns. There are a bunch of examples of it running different kinds of scripts (sql or bash) at different intervals (weeks, months, days) [16:48:31] it's a pretty cool tool, I'll paste you some example configs for a layout similar to yours [16:49:07] language team does some simple sql daily: https://github.com/wikimedia/analytics-limn-language-data/tree/master/language (the yaml file is the config, sql are the script templates, notice how they take wiki as param) [16:49:25] actually, just start with that and ask me if you have Qs [16:49:38] Neat [16:50:24] milimetric: I guess I'd need my own repository for this, and run it on my user on stat1003? [16:52:45] Analytics-General-or-Unknown, Community-Advocacy, Wikimedia-Extension-setup, Wikipedia-iOS-App-Product-Backlog: enable Piwik on ru.wikimedia.org - https://phabricator.wikimedia.org/T91963#1944321 (Milimetric) > Please update https://wikimediafoundation.org/wiki/Privacy_policy/FAQ#cookieFAQ with th... [17:02:07] joal: standup! [17:02:33] sorry! forgot [17:03:43] Analytics-Kanban: Foundation-only Geowiki stopped updating - https://phabricator.wikimedia.org/T106229#1944361 (Milimetric) a:Milimetric [17:04:48] milimetric: can you restart piwik? it's kaput [17:12:02] Analytics-Tech-community-metrics, Developer-Relations, DevRel-January-2016, developer-notice: Check whether it is true that we have lost 40% of (Git) code contributors in the past 12 months - https://phabricator.wikimedia.org/T103292#1944375 (Aklapper) >>! In T103292#1933663, @Qgil wrote: > Thank yo... [17:12:23] Analytics-Kanban: Foundation-only Geowiki stopped updating - https://phabricator.wikimedia.org/T106229#1944376 (jcrespo) s1, s2, s3, s4, s5, s6, and s7 -analytics-slave is only a DNS, there are only 2 analytics slaves (as physical machines), that I know of: s1 and s2 (db1047) all others (dbstore1002) (It is... [17:24:05] milimetric: Hm, it looks like reportgenerator/reports is what I need to modify to get the updates to run, right? [17:28:53] MarkTraceur, hey can I help you with reportupdater? [17:29:16] mforns: Matter of fact you can! [17:29:28] I have the SQL all copied over, I think I just need to make these PHP files [17:30:03] MarkTraceur, mmm I don't think you need any php... [17:30:12] Oh, the language one has some PHP stuff [17:30:32] you're right, but that is not needed I think [17:30:41] do you have your own repo? [17:31:13] Not on the internet yet, no [17:31:25] But I have it cloned from github and could fork to show you what's happening so far [17:32:30] aha [17:33:05] mforns: https://github.com/MarkTraceur/analytics-limn-language-data [17:33:10] Should have changed the name [17:33:11] Oh well [17:33:29] I'll break that link later. [17:33:56] what is what you want to achieve? just update sql-generated reports periodically? [17:34:24] Analytics, MediaWiki-extensions-WikimediaEvents, The-Wikipedia-Library, Wikimedia-General-or-Unknown, Patch-For-Review: Implement Schema:ExternalLinkChange - https://phabricator.wikimedia.org/T115119#1944406 (Sadads) @Beetstra mentioned the project in T121094 . Wondering if there is any action... [17:35:52] MarkTraceur, if so, you just need the config.yaml file and the query files. [17:36:23] mforns: Well, I should probably also generate the initial reports, but yeah [17:36:51] mforns: Also, I think I asked milimetric about this last time, but they're monthly reports, which is hopefully supported because I just dumped "monthly" in where it used to say "daily" [17:36:55] MarkTraceur, sure, reportupdater will do that for you [17:37:13] * mforns looks [17:38:30] MarkTraceur, yes, it supports monthly granularity and frequency. However it's "months", not "monthly" [17:38:55] here's a documentation: https://wikitech.wikimedia.org/wiki/Analytics/Reportupdater [17:40:52] mforns: Right, I followed the format that was in the initial config file, so days became months, sorry about that [17:41:14] Looks like the documentation doesn't say "months" for frequency [17:41:49] MarkTraceur, you're right! Monthly support was added recently, and the docs are outdated, thanks for spotting that :] [17:42:06] mforns: For "months", should "starts" be 2004-01 or 2004-01-01? [17:42:32] it should be 2004-01-01 [17:42:47] K, fixing [17:44:36] (CR) Joal: "Hi Nuria," [analytics/refinery/source] - https://gerrit.wikimedia.org/r/264940 (https://phabricator.wikimedia.org/T122651) (owner: Joal) [17:44:48] fixed the docs, thx [17:47:10] OK mforns, fixed...will try to run it now [17:47:41] MarkTraceur, have you done the ssh tunneling? [17:47:55] Tunneling? Like, to stat1003? [17:48:00] yes [17:48:07] Yeah, I'm a deployer, I know what's what [17:48:17] ok ok [17:48:23] Besides which I worked on the past two failed attempts at multimedia metrics :P [17:49:57] Analytics: Make Dashiki get pageview data from pageview API - https://phabricator.wikimedia.org/T124063#1944457 (Milimetric) NEW [17:50:40] nuria: just got done with post standup stuff. I [17:50:57] *I'll restart mysql, but piwik by itself is definitely not keeping up here [17:51:56] nuria: ok, restarted. But don't make the same mistake I made and try to look at the dashboard, that's what kills the server :) [17:52:02] mforns: Do I leave this running in a screen, or should I cron it? [17:52:14] milimetric: ay ay , well we will worry about that later. [17:52:21] nuria: and without trying to pull up the dashboards, it seems it's fine to just record the events [17:52:27] The command documentation says "Periodically execute" which makes me think the former [17:52:29] milimetric: k [17:52:48] like, I looked at max(timestamp) and it keeps growing. There may be loss, but we can know for sure if we compare with webrequest [17:52:51] MarkTraceur, you should cron it, it's not a deamon, just runs once and stops. [17:53:00] OK [17:53:09] mforns: Is it worth it to run it once outside of cron, too? [17:53:13] MarkTraceur, we have our other instances running every hour [17:53:17] *nod* [17:53:49] I have a strange feeling that git-clone will not work with github on the cluster [17:53:55] Yeah no [17:54:00] * MarkTraceur muses [17:54:36] MarkTraceur: we have a puppet setup that runs this thing on a cron [17:55:01] milimetric: That might be superior [17:55:04] if you put it in gerrit under analytics/limn-{something}-data and name the folder where config.yaml is {something}, everything else will just work [17:55:16] I can make that repo for you in gerrit if you want [17:55:22] OK, sure [17:55:24] thx milimetric [17:55:32] So it's limn-multimedia-health-data [17:55:49] And it's https://github.com/MarkTraceur/analytics-limn-language-data currently, which should be importable [17:55:49] oh, I think I already did: https://gerrit.wikimedia.org/r/#/admin/projects/analytics/limn-multimedia-data [17:56:00] Uhhhh I guess changing the directory name could work [17:56:11] oh yeah, MarkTraceur https://github.com/wikimedia/analytics-limn-multimedia-data/tree/master/multimedia [17:56:14] milimetric, MarkTraceur, I have a meeting now, please let me know if I can help further :] [17:56:25] I remember better now, we already did this a few months ago :) [17:56:33] Only kinda [17:56:42] I flaked because I had other stuff that became priority [17:56:43] my memory either works after I remember, or I am good at brain-washing myself [17:56:54] np at all, no judgement, just <3 [17:56:55] :) [17:57:05] OK, directory renamed, that repo will work now... [17:57:23] but yeah, check out what's in there, I had set up a sample for you as well [17:57:24] Now I have to decide between making ostriches angry or just plopping everything into it [17:57:48] plop, do whatever and I'll merge and we can set up the cron through puppet [17:57:52] just let me know when to do it [17:58:00] milimetric: Hm, from_timestamp and to_timestamp were not in the language example you gave me, should I add them? [17:58:15] I have https://github.com/MarkTraceur/analytics-limn-language-data/blob/master/multimedia/uploads.sql [17:58:21] MarkTraceur: yes, that's what takes in the timespan that your report should run from/to [17:58:45] MarkTraceur: lemme find an actually working example of that [17:59:43] MarkTraceur: ok, this one has time params and even an additional "editor" param which is not hardcoded so you can see how to do that: [17:59:44] https://github.com/wikimedia/analytics-limn-edit-data/blob/master/edit/sessions.sql [17:59:56] and the config: https://github.com/wikimedia/analytics-limn-edit-data/blob/master/edit/config.yaml#L57 [18:00:20] MarkTraceur: notice the "timeboxed: true" part which is what tells it the script understands {from_timestamp} and {to_timestamp} [18:01:05] you don't have to use both, some people just rely on one or the other depending on how they run their rerport. But basically reportupdater will use the from_timestamp as the key in your output file [18:01:28] so if it misses a day, it'll run again with from_timestamp = that missed day and to_timestamp = that missed day + your interval [18:03:18] mforns: coming to meeting with madhuvishy [18:03:19] ? [18:03:35] Neat, OK, updating [18:04:19] nuria, I'm in the hengout! [18:04:35] nuria, madhuvishy, are you in the batcave? [18:04:48] mforns: no - meeting link in calendar [18:04:56] madhuvishy, me too... :] [18:05:02] mforns: https://plus.google.com/hangouts/_/wikimedia.org/madhu-s?authuser=0 [18:05:39] (PS1) MarkTraceur: Add files stolen from language [analytics/limn-multimedia-data] - https://gerrit.wikimedia.org/r/264977 [18:05:47] There you go mili|lunch [18:07:36] Ping me if I have to do anything else, I'm going to stare at a halfak python script for a while and try to figure out how I'm going to squelch it into useful data [18:07:46] I mean, more useful data [18:07:54] It's already pretty freakin' cool. [18:08:04] * halfak reads scrollback [18:08:24] Which python script are you looking at? [18:09:21] halfak: You wrote one for detecting additions and removals of file links on pages [18:09:33] Which is super useful raw data that I have to distill, like, a lot. [18:09:44] And run it on other wikis that aren't enwiki [18:10:53] Gotcha! [18:11:05] I use that as an example for mwxml these days. See https://tools.wmflabs.org/paws/public/EpochFail/examples/mwxml.py.ipynb [18:11:29] Oh cool [18:11:33] I did some simplifications and demoed it on nlwiki even though they probably localize the name for "File/Image" [18:11:47] *nod* [18:11:47] Just because they have fewer dump files and that looks better in the example. [18:12:05] Oh, shoot, that's probably why I decided it would be Bad News™ to try it on non-English wikis [18:12:06] Crap. [18:12:22] MarkTraceur, either look up the localization or make a request to the API [18:12:40] Right, it's fine, but still more painful than it should be [18:12:46] https://en.wikipedia.org/w/api.php?action=query&meta=siteinfo&siprop=namespaces [18:12:58] I guess I could use HTML dumps instead...but that would take even longer [18:13:02] How could it be any other way? [18:13:13] Oh yeah. Render the content. [18:13:22] (and yeah, I know siprop, I wrote the nasty namespace handling in Parsoid) [18:13:34] Oh! I forgot that the dump has localized namespace names! [18:13:46] Hm? [18:14:00] OK, here's a question I should know the answer to [18:14:07] http://pythonhosted.org/mwxml/iteration.html#mwxml.SiteInfo [18:14:23] If you're linking to a Commons image on nlwiki, can you use the localized file namespace name? [18:14:30] Commons doesn't have that localization [18:14:43] MarkTraceur, seems like that would be required. [18:14:54] I can't imagine it not resolving that localized name. [18:15:00] True [18:15:12] It doesn't even know the link is an image without localizing to the namespace first. [18:15:18] I guess that kinda jives with how remote repos work, but gosh, I hate remote repos [18:15:24] ha! [18:34:00] ottomata: https://github.com/linkedin/Burrow/issues/4#issuecomment-172944046 [18:46:47] joal: fyi, mobile -> text discussion is happening in #wikimedia-traffic [18:47:24] ok, thx ottomata [18:49:06] Analytics-Kanban: Projections of cost and scaling for pageview API. {hawk} [8 pts] - https://phabricator.wikimedia.org/T116097#1944709 (JAllemandou) [18:51:23] How often does the magic rsync happen these days on stat1003 [18:59:13] Analytics-Kanban, DBA, Patch-For-Review: EL replication having issues since at least January 11th - https://phabricator.wikimedia.org/T123634#1944742 (Neil_P._Quinn_WMF) FYI, I ended up killing that query (the one which started `INSERT INTO editor_month_global`) Sunday morning as it still hadn't comple... [19:07:06] nice madhuvishy :) [19:07:23] Analytics-Kanban: Projections of cost and scaling for pageview API. {hawk} [8 pts] - https://phabricator.wikimedia.org/T116097#1944812 (JAllemandou) Had a talk with @gwicke about cassandra read latency problem (disk seek takes long time on rotating disks). He suggests we replace the rotating disks with SSDs (... [19:07:45] ottomata: I'm lost in traffic chan :) [19:09:09] joal: heh, i'm not relaly folloowing [19:09:16] we only need to know when they are done, ja? [19:09:25] hm, probably [19:09:57] joal, merge, ja? [19:09:58] https://gerrit.wikimedia.org/r/#/c/264870/ [19:10:04] ottomata: hi [19:10:10] you pinged yesterday? [19:10:22] oh ja [19:10:22] about [19:10:24] ottomata: merging ! [19:10:50] ori about: https://phabricator.wikimedia.org/T123954 [19:11:02] (CR) Joal: [C: 2 V: 2] "Merging after multiple reviews." [analytics/refinery] - https://gerrit.wikimedia.org/r/264870 (https://phabricator.wikimedia.org/T122651) (owner: Ottomata) [19:11:05] wanted to know about you and aaron's cross DC use case [19:12:47] ottomata: I take a break to have diner [19:12:59] Will come back after, see if there is anything I can help on [19:13:24] Analytics, Analytics-Cluster, EventBus: Update MirrorMaker in Kafka .deb and puppet module - https://phabricator.wikimedia.org/T124077#1944861 (Ottomata) NEW a:Ottomata [19:13:25] k cool [19:14:44] Analytics-Tech-community-metrics, DevRel-January-2016: Make GrimoireLib display *one* consistent name for one user, plus the *current* affiliation of a user - https://phabricator.wikimedia.org/T118169#1944870 (Aklapper) [19:14:46] Analytics-Tech-community-metrics, Developer-Relations, DevRel-February-2016: Who are the top 50 independent contributors and what do they need from the WMF? - https://phabricator.wikimedia.org/T85600#1944869 (Aklapper) [19:15:54] Analytics-Cluster, EventBus, Services, operations: Investigate proper set up for using Kafka MirrorMaker with new main Kafka clusters. - https://phabricator.wikimedia.org/T123954#1944872 (Ottomata) [19:16:28] ottomata: cross-DC messaging _is_ the use-case [19:16:29] mforns: i'm going to deploy your EL change to beta [19:16:33] for Aaron's requirements [19:16:40] ottomata, cool [19:16:54] ottomata, do you want me to co-deploy? [19:17:01] ottomata: I'll ask Aaron to chime in [19:17:06] mforns: actually, if you are here and want to just do it, go right ahead :) [19:17:09] ori: cool, right. so, currently we are looking at naming topics based on DC [19:17:18] and using mirror maker to mirror them between each DC [19:17:19] so [19:17:23] ottomata, ok will do [19:17:35] topic: purge_event.codfw [19:17:40] producers in codfs [19:17:50] ottomata: wait one sec, asking aaron to join [19:17:51] not sure if he's here [19:17:53] ok [19:17:58] ori we can move to #ops [19:18:07] or anywhwere [19:18:22] or he can just chime in on ticket [19:18:23] no hurry [19:19:15] Analytics-Tech-community-metrics, Developer-Relations, DevRel-February-2016: Who are the top 50 independent contributors and what do they need from the WMF? - https://phabricator.wikimedia.org/T85600#1944883 (Aklapper) Added {T118169} as dependency as [[ http://korma.wmflabs.org/browser/top-contributor... [19:19:16] seems he's not here, so go on [19:19:21] the recap is useful [19:19:23] (to me) [19:19:25] Analytics-Tech-community-metrics, DevRel-January-2016: Many profiles on profile.html do not display identity's name though data is available - https://phabricator.wikimedia.org/T117871#1785578 (Aklapper) [19:21:04] ottomata, what's the equivalent of tin for eventlogging beta deployment (for git deploy)? [19:21:30] wikimedia/mediawiki-extensions-EventLogging#523 (wmf/1.27.0-wmf.11 - 38e0d09 : Dan Duvall): The build has errored. [19:21:31] Change view : https://github.com/wikimedia/mediawiki-extensions-EventLogging/commit/38e0d093f86e [19:21:31] Build details : https://travis-ci.org/wikimedia/mediawiki-extensions-EventLogging/builds/103424342 [19:23:07] ja so [19:23:22] if we want events from both DCs in both DC kafka clusters [19:23:32] (PS2) MarkTraceur: Add files stolen from language [analytics/limn-multimedia-data] - https://gerrit.wikimedia.org/r/264977 (https://phabricator.wikimedia.org/T111793) [19:23:39] we need to distinctly name the topics after the DC [19:23:43] (or whatever) [19:23:53] so that MirrorMaker doesn't produce in a feedback loop [19:24:04] all MirrorMaker does is consume from source clusters and produce to dest cluster [19:24:18] so if we have MirrorMaker instances in both DCs consuming and producing to the same topic in a DC [19:24:36] then a message would be produced in an infinite loop [19:24:52] the way to get around it is to name topics distinctly [19:25:06] so that messages produced in codfw are in a different topic than messages produced in eqiad [19:25:24] MirrorMaker instance in each DC would then consume from the other and produce to the local cluster [19:25:25] so [19:25:33] topic: purge_event.codfw [19:25:38] producers and codfw would write there [19:26:04] and MirrorMaker in eqiad would consume that topic from codfw and write to the purge_event.codfw topic in the cluster in eqiad [19:26:12] so, if you want all events from both DCs [19:26:16] you need to consume both topics [19:26:31] the other way to do this (and the way LInkedIn recommends) [19:26:42] is to run local and aggregate clusters in each DC [19:26:48] producers always produce to local cluster [19:27:03] and MirrorMaker in each DC consumes from all local clusters in all DCs, and writes to the DC's aggregate cluster [19:27:22] the aggregate clusters would only have events produced by MirrorMaker [19:27:35] if a consumer needs all events in all DCs, they'd consume from their DC's aggregate cluster [19:27:41] buuuut, that requires more hardware and more clusters :/ [19:29:26] ori ^^ :) [19:29:37] ottomata: still wrapping my head around this, just a moment [19:30:37] ottomata: for purging stuff either way is fine, I guess the later is easier to manage daemons for rather than having ones that either read X channels or puppetizing X daemons per cache node [19:31:50] ottomata: how much additional hardware would it require? [19:34:07] ottomata, ^ what's the equivalent of tin for eventlogging beta deployment (for git deploy)? [19:35:39] mforns: deployment-bastion [19:35:47] ottomata, thx [19:35:49] ori, hardware, uh well, +1 cluster in each DC [19:36:20] aggregate cluster is expected to have more load, as consumers should use it, and it will have data from all DCs [19:36:40] but, for our initial use cases i don't htink we'd need more than the same hw we already ordered [19:36:58] (I invited paravoid) [19:37:01] sec, getting the backlog [19:37:06] wanted his opinion as well [19:37:07] ori, AaronSchulz: i think we should just go with the DC topic names for now, and do it that way [19:37:23] mili|lunch: OK this one's weird, http://datasets.wikimedia.org/limn-public-data/metrics/multimedia-health/new-uploaders/ shows a 1.8k file commonswiki.tsv but every time I try to download it with chrome, it's a 0-byte file [19:37:27] 1. we can do it now [19:37:27] 2. it might be handy to be able to consume data for just one DC instead of having it all mixed up [19:37:28] for purges, we could easily change it later [19:37:40] Cocking up my dashiki testing [19:37:43] yeah, i think it would all be fairly easy. it is more complicated for consumers, depending on your client implementation [19:37:49] because you'd have to consume from more than 1 topic [19:37:51] to get all events [19:38:04] I guess I'll kill some caches, sec [19:38:05] but, some consumers abstract that away just fine [19:38:25] paravoid: https://dpaste.de/jeN4/raw [19:38:48] OK yeah I'm an idiot [19:38:53] But wow that was an aggressive cache [19:39:23] ottomata: OTOH, cross-DC purge propagation is one of those pieces of infrastructure that should work _really_ well, and is worth the investment [19:39:34] migrating later could be more difficult than we imagine right now [19:39:49] so I'm inclined to do it right from the get-go [19:40:06] ori: https://engineering.linkedin.com/kafka/running-kafka-scale [19:41:29] we've discussed this a bit before ottomata, right? [19:41:44] from the little I know about kafka, I think it's where we should be [19:41:57] the fact that this is how LinkedIn (who wrote Kafka) set things up is pretty important, IMO. We'd be less likely to hit unexpected snags [19:42:13] is that eventbus-specific though? [19:42:32] are you still planning to keep the webrequest cluster(s) separate? [19:46:56] ottomata: ^ [19:48:28] Analytics: Move vital signs to its own instance {crow} [5 pts] - https://phabricator.wikimedia.org/T123944#1944976 (madhuvishy) p:Triage>Normal [19:48:32] (PS3) MarkTraceur: Add files stolen from language [analytics/limn-multimedia-data] - https://gerrit.wikimedia.org/r/264977 [19:49:07] sorry was afk for a sec [19:49:10] paravoid: , ori, yes that is separate [19:49:20] since we do cross DC production for webrequest data [19:49:22] (PS1) MarkTraceur: Add queries for per-tool uploads from UW and CWU [analytics/limn-multimedia-data] - https://gerrit.wikimedia.org/r/264996 (https://phabricator.wikimedia.org/T111793) [19:49:23] directly to analytics cluster [19:49:25] that is separate [19:49:26] why not, thouigh? [19:49:41] why not what? [19:49:41] if we're planning for the long-term, that needs to be part of the equation, right? [19:49:55] +1 [19:49:56] oh [19:50:01] then sure i'm all for it [19:50:03] I mean it's pretty unfortunate that we may lose logs for the duration of the codfw switchover [19:50:32] so ja, here's the ideal setup i think. [19:50:39] 1 big analytics cluster in just eqiad (the one we have) [19:50:43] in each DC [19:50:54] 2 small-mediumish (whatever we may need) clusters [19:50:55] Analytics: Make Dashiki get pageview data from pageview API - https://phabricator.wikimedia.org/T124063#1944995 (madhuvishy) [19:50:57] Analytics: Move vital signs to its own instance {crow} [5 pts] - https://phabricator.wikimedia.org/T123944#1944994 (madhuvishy) [19:51:03] local and aggregate [19:51:22] mirrormaker in each DC consumes from all DC-'local' clusters into the DC's aggregate cluster [19:51:32] why analytics in just eqiad? [19:51:41] kafka specifically, I mean [19:52:33] hm, well, actually you are right, i'm not certain. i guess if all clusters were beefy high throughput clusters, it'd be ok [19:52:40] and we woudln't need a specific analytics kafka cluster [19:53:01] it is possible that if realtime analytics becomes more prevalent, it could be a source of load on a kafka cluster [19:53:07] so that would be a reason to have a separate analytics cluster [19:53:44] if there was a separate analytics kafka cluster, it would use mirrormaker to consume from the aggregate clusters in each DC [19:54:16] the simpler layout is to not make analytics kafka special though [19:54:24] to have just 2 clusters in each DC, local and aggregate [19:54:33] analytics uses would consume from the aggregate cluster, just like everyone else [19:55:23] hm, in caching only DCs, we would only need the 1 'local' cluster [19:55:36] (if there are no consumers there...oh but i guess there will be?) [19:55:44] there will be [19:55:48] aye, nm then [19:56:06] so, ja, if analytics does not need to be separate, then the ideal setup is 2 beefy clusters per DC [19:56:16] Analytics: Move vital signs to its own instance {crow} [5 pts] - https://phabricator.wikimedia.org/T123944#1945019 (madhuvishy) Vital Signs currently gets pageviews data from https://github.com/wikimedia/analytics-aggregator-data and https://github.com/wikimedia/analytics-aggregator-projectview-data. These g... [19:56:31] if analytics does need to be separate, than at least +1 for eqiad, maybe another for codfw [19:56:33] we're not that far off from that, are we [19:56:37] but id ont' think the codfw analytics is necessary [19:56:42] if we have the aggregate clusters [19:56:42] we have two clusters in eqiad and one in codfw [19:56:54] hardware-wise, I mean [19:57:14] yessssss, but the hw specs are different for those [19:57:20] Analytics, Reading-Web, Wikipedia-iOS-App-Product-Backlog: As an end-user I shouldn't see non-articles in the list of trending articles - https://phabricator.wikimedia.org/T124082#1945028 (BGerstle-WMF) NEW [19:57:24] analytics is 6 node x 12 disks [19:57:33] right [19:58:22] if we want to do the right thing sooner, we'd not worry about webrequest data yet and add +1 cluster to each DC [19:58:26] for local/aggregate setup [19:58:53] we'd then have a total of 5 clusters: analytics-eqiad, main-eqiad, aggregate-eqiad, main-codfw, aggregate-codfw [19:59:24] ugh, I missed it when you bumped it into 6 [19:59:28] this is really not well distributed [19:59:33] 3 out of 6 are in row A [19:59:36] 5! [19:59:46] if we lose row A, we're going to lose quorum :/ [19:59:47] oh nodes? [19:59:52] ?? in row A? [19:59:54] checking... [20:00:10] 1001, 1012, 1013 [20:00:26] 1012/1013 are on the same rack even, as are 1018/1020 [20:00:38] and 1014/1022 [20:00:40] Analytics: Look through wikimetrics/scripts and clean them up {dove} - https://phabricator.wikimedia.org/T123956#1945050 (madhuvishy) Spoke to @milimetric about this - For now we'll remove the following scripts: dependencies, install, find_todos, deploy, populate_tags, update_and_restart. These are supersed... [20:00:42] oh no [20:00:43] paravoid [20:00:47] 1001 is not in the analytics cluster [20:01:13] https://github.com/wikimedia/operations-puppet/blob/production/hieradata/common.yaml#L338 [20:01:18] Analytics, Reading-Web, Wikipedia-iOS-App-Product-Backlog: As an end-user I shouldn't see non-articles in the list of trending articles - https://phabricator.wikimedia.org/T124082#1945061 (BGerstle-WMF) p:Triage>High [20:01:26] anyway, offtopic, sorry! [20:01:28] hhe [20:01:43] analytics cluster has 2 nodes in each row A C and D [20:02:22] so yeah, we don't really need 2 sets of clusters for DC based on our current needs [20:02:27] indeed [20:02:33] but I think it's worthwhile to be thinking more long-termish [20:02:40] and if we do, we should probably take the webrequest cluster into account [20:02:47] * milimetric need to run to the doctor :( I'll be back shortly hopefully. [20:02:49] Analytics-Kanban, DBA, Patch-For-Review: EL replication having issues since at least January 11th - https://phabricator.wikimedia.org/T123634#1945064 (jcrespo) For those kind of questions "general consultancy/help", it is quicker if you schedule a chat/meeting. I am trying to organize a session for dev... [20:03:07] I'll need to run very soon too -- we could move this into phab afterwards perhaps? [20:03:54] ja, paravoid here: https://phabricator.wikimedia.org/T123954 [20:04:08] awesome [20:06:10] Analytics-Cluster, EventBus, Services, operations: Investigate proper set up for using Kafka MirrorMaker with new main Kafka clusters. - https://phabricator.wikimedia.org/T123954#1945096 (Krinkle) >>! In T123954#1944108, @Ottomata wrote: > I think I'd prefer suffixing, but I'm not sure. E.g. `med... [20:06:12] mforns: Would you mind terribly reviewing and merging my patch(es) to the multimedia limn data repo? [20:06:19] mforns: https://gerrit.wikimedia.org/r/264977 [20:08:37] !log deployed eventlogging to deployment-eventlogging03 with removal of mysql consumer batch [20:08:40] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log, Master [20:10:24] Analytics-Cluster, EventBus, Services, operations: Investigate proper set up for using Kafka MirrorMaker with new main Kafka clusters. - https://phabricator.wikimedia.org/T123954#1945176 (Ottomata) `main-eqiad` is the name (I have chosen :?) for the Kafka cluster, so I'd like to keep it consistent... [20:10:56] ottomata, should I deploy EL to deployment-eventlogging04, too? [20:11:35] sure its fine, that is running the beta eventbus [20:11:36] hey ottomata [20:11:37] won't matter [20:11:53] any news on mobile/text? [20:11:53] hey joal [20:12:06] let's ask! [20:12:12] :)P [20:12:17] ottomata, ok [20:12:26] Also, interesting discussion about kafka [20:15:36] Analytics-Cluster, EventBus, Services, operations: Investigate proper set up for using Kafka MirrorMaker with new main Kafka clusters. - https://phabricator.wikimedia.org/T123954#1945236 (Krinkle) >>! In T123954#1945176, @Ottomata wrote: > `main-eqiad` is the name for the Kafka cluster, so I'd like... [20:19:06] MarkTraceur, looking at it [20:21:44] ottomata: can you give your opinion on https://phabricator.wikimedia.org/T116097 ? [20:22:06] ottomata: Let's push my long night work to tomorrow :) [20:22:24] Analytics, Reading-Web, Wikipedia-iOS-App-Product-Backlog: As an end-user I shouldn't see non-articles in the list of trending articles - https://phabricator.wikimedia.org/T124082#1945327 (BGerstle-WMF) For example top.hatnote.com uses a meta query to filter out a site's main page, "-", and anything... [20:25:26] joal: k [20:26:38] also ottomata, thanks for putting me in the kafka inter-dc discussion ,super interesting :) [20:27:00] sho thang [20:27:50] you're my slang teacher ottomata :) [20:28:43] milimetric: Do you think we should prioritize this https://phabricator.wikimedia.org/T124082 ? [20:29:32] Analytics-Kanban: Projections of cost and scaling for pageview API. {hawk} [8 pts] - https://phabricator.wikimedia.org/T116097#1945406 (Ottomata) Interesting. I suppose we could fit those SSDs into the nodes we currently use, but as Kevin notes, they are soon going to be out of warrantee. I think it'll be v... [20:31:09] milimetric: I forgot you were off to the doctor ... My bad. I'm off for tonight, let's discuss the pageview API ticket tomorrow :) [20:31:17] Good night a-team, see you tomorrow ! [20:31:26] bye joal good night! [20:32:14] nighters! [20:32:34] o/ [20:33:52] Analytics, Reading-Web, Wikipedia-iOS-App-Product-Backlog: As an end-user I shouldn't see non-articles in the list of trending articles - https://phabricator.wikimedia.org/T124082#1945440 (Ottomata) Aye, I had to do this as well in https://gerrit.wikimedia.org/r/#/c/225485/1/refinery-job/src/main/sca... [20:34:08] Analytics-Tech-community-metrics: SCM project 'core' contains Pywikibot and MediaWiki - https://phabricator.wikimedia.org/T123808#1945444 (jayvdb) p:High>Unbreak! [20:37:28] Analytics-Kanban: Dedicated and/or automated Wikimedia pageviews API project/tag in Phabricator Maniphest [1 pts] - https://phabricator.wikimedia.org/T119151#1945469 (madhuvishy) [20:38:18] Analytics-Kanban: Dedicated and/or automated Wikimedia pageviews API project/tag in Phabricator Maniphest [1 pts] - https://phabricator.wikimedia.org/T119151#1819445 (madhuvishy) Made a task here - https://phabricator.wikimedia.org/T124086 [20:43:47] MarkTraceur, I have to leave now, I work from Europe, and it's kinda late for me, are you OK with me finishing your review tomorrow morning? [20:45:30] mforns: sure! Maybe Dan can finish if he gets back too [20:59:04] (CR) Mforns: Add files stolen from language (4 comments) [analytics/limn-multimedia-data] - https://gerrit.wikimedia.org/r/264977 (owner: MarkTraceur) [21:00:06] MarkTraceur, OK, I left some partial comments in case someone else wants to finish the review. Otherwise, I'll finish that tomorrow. thx! [21:00:24] bye a-team! see you tomorrow! [21:01:04] byeo [21:03:17] Analytics: Move vital signs to its own instance {crow} [5 pts] - https://phabricator.wikimedia.org/T123944#1945640 (Nuria) @madhuvishy, @milimetric: dashiki also displays legacy pageviews which are only available via the github depot system. Given that we want to keep those around (since we do not have pagev... [21:08:00] Analytics, Discovery, Reading-Infrastructure-Team: Determine proper encoding for structured log data sent to Kafka by MediaWiki - https://phabricator.wikimedia.org/T114733#1945694 (Nuria) >This data pipeline is only intended to feed a single Hive/Hadoop table that will be processed using additional Hiv... [21:09:27] Analytics-Kanban, DBA, Patch-For-Review: Pending maintenance on the eventlogging databases (db1046, db1047, dbstore1002, other dbstores) - https://phabricator.wikimedia.org/T120187#1945712 (Nuria) @jcrespo: Thursday sounds good. We will announce it cc @ottomata to confirm that we are not deploying any... [21:10:06] (PS4) MarkTraceur: Add files stolen from language [analytics/limn-multimedia-data] - https://gerrit.wikimedia.org/r/264977 [21:11:51] (PS2) MarkTraceur: Add queries for per-tool uploads from UW and CWU [analytics/limn-multimedia-data] - https://gerrit.wikimedia.org/r/264996 (https://phabricator.wikimedia.org/T111793) [21:12:23] milimetric: OK, mforns reviewed it a bit, if you can review the rest that'd be super duper [21:16:02] Analytics, Discovery, Reading-Infrastructure-Team: Determine proper encoding for structured log data sent to Kafka by MediaWiki - https://phabricator.wikimedia.org/T114733#1945745 (Ottomata) Hm, just a thought. 450M per day is about 50 per second, ja? If we blacklist this schema from going into MySQL... [21:19:26] MarkTraceur: if you could info to https://wikitech.wikimedia.org/wiki/Analytics/Dashiki that we might be missing (or gotchas for another users) it will be great. thank you [21:23:42] ottomata: are you planning on deploying the batching-removing code to EL ? [21:24:44] not today, was going to let it run in beta as is [21:24:50] don't really have a plan for it though [21:30:45] ottomata: i was asking since jaime wants to do teh toku db maintance and stop consumer for the weekend [21:30:51] ottomata: mysql consumer that is [21:31:17] hm for the whole weekend, eh? [21:31:18] heh [21:31:22] i guess we should deploy this thing first? [21:31:30] sounds risky [21:31:48] nuria: if all is well on beta tomorrow, i'll ask mforns if we can deploy it together [21:32:06] ottomata: then, should we postpone maintenance for this thursday? [21:33:38] well, its risky to combine the two, as there could be unknowns in running the new code in prod [21:34:07] but, it would probably be better to use the queueless code to backfill from kafka when the downtime is done [21:38:17] I what point should I start to worry that every conversation I have about moving data from MediaWiki to HDFS brings up a new way to do it? ;) [21:38:44] nuria: It was mostly fine for me, I just asked milimetric because I knew (from our past conversation) that he had opinions on what we should be using [21:39:15] bd808: Still getting at least 6 hours of sleep every night? Probably fine. [21:39:22] ottomata: are there any really strong reasons *not* to use Avro in the same way that Discovery is doing for the search data? [21:39:30] MarkTraceur: ok, thank you, it just a pointer to our docs cause you know we love contributions and we would like to make things like dashiki layouts as self serve as possible [21:39:41] bd808: not to use it? [21:39:47] bd808: (sorry to chime in) [21:39:58] This was actually pretty fantastic, given that I got it done, can expand it myself, and don't need to come back and maintain it [21:40:19] nuria: chimes welcome. This is about the things I said in https://phabricator.wikimedia.org/T114733#1942809 [21:40:35] bd808: complexity that i can think of, really getting the whole avro workflow took us (analytics and search) quite some cycles [21:41:02] and that may not be repeatable now? [21:41:07] bd808: jaja [21:41:45] bd808: hopefully we learned it all but notice on dcausse's doc that schema evolution in avro is quite less trvial than we thought [21:41:49] *trivial [21:41:56] bd808: there is manual work you have to do [21:42:08] k. I just don't want to invent new things if possible since this was supposed to be a Q2 goal (based on the assumption that Discovery had "solved" the problem in Q1) [21:42:48] I naively thought it take me a week in October/November to get data flowing [21:43:07] bd808: well, we thought it will be couple of days to get avro going jaja [21:44:09] Right, and I have a general grasp on the problems that were found and worked around. I'm just worried about suggestions that I switch to another path because I'm not sure these other paths are an more travelled than avro [21:44:31] bd808: anyways, cc ottomata i think you can do avro and use our learnings, using eventlogging has a lot less friction when it comes to schema evolution but querying data is not doable from hive directly [21:46:05] *nod* so the trade today is easy schema evolution vs an extra ETL step to make the data usable by hive [21:46:49] bd808: I think we can only learn by trying, so go for avro 1st, do a tryout and should things not work you can fall back on EL [21:47:12] nuria: sweet. That was just about what I was going to suggest too. :) [21:47:19] cc ottomata [21:47:23] so he is aware [21:47:31] ah, i guess he will see ticket [21:47:39] ja that is fine with me i think [21:48:19] Analytics-Tech-community-metrics: SCM project 'core' contains Pywikibot and MediaWiki - https://phabricator.wikimedia.org/T123808#1945858 (Aklapper) @jayvdb, Thanks for finding this and raising this! And I think you are indeed right. `Something-else-core` might be `oojs/core`. To get some data to compare th... [21:49:02] Analytics-Tech-community-metrics, DevRel-January-2016: Statistics for SCM project 'core' mix pywikibot/core, mediawiki/core and oojs/core - https://phabricator.wikimedia.org/T123808#1945861 (Aklapper) [21:52:30] Analytics, Discovery, Reading-Infrastructure-Team: Determine proper encoding for structured log data sent to Kafka by MediaWiki - https://phabricator.wikimedia.org/T114733#1945865 (bd808) @nuria and I talked this out on irc a bit and had these thoughts: * Schema migration (adding/removing fields) using... [22:19:10] (CR) Milimetric: [C: 2 V: 2] Add files stolen from language [analytics/limn-multimedia-data] - https://gerrit.wikimedia.org/r/264977 (owner: MarkTraceur) [22:20:22] (CR) Milimetric: [C: 2 V: 2] "I'll comment here with the link to the puppet that needs to be merged for this to be crontabbed on stat1003." [analytics/limn-multimedia-data] - https://gerrit.wikimedia.org/r/264996 (https://phabricator.wikimedia.org/T111793) (owner: MarkTraceur) [22:23:21] ottomata: if you don't mind, this is just another reportupdater cron job: https://gerrit.wikimedia.org/r/#/c/265152/ [22:23:40] (CR) Milimetric: "The puppet that needs to be merged for this to run is: https://gerrit.wikimedia.org/r/#/c/265152/" [analytics/limn-multimedia-data] - https://gerrit.wikimedia.org/r/264996 (https://phabricator.wikimedia.org/T111793) (owner: MarkTraceur) [22:24:07] milimetric: that's all ready to go with repos and stuff? [22:24:14] ottomata: yes [22:27:01] milimetric: done and puppet run. [22:27:07] generate_multimedia_limn_public_data]/ensure: created [22:27:39] thx! [22:32:26] Woot woot [22:32:44] milimetric: So we have scripts running, like, right now? :D [22:34:19] MarkTraceur: since you set up the frequency as "months" it would run monthly. But since the reports need to catch up, I think the next run of the cron will pick them up and run all the old months. [22:34:32] OK cool, is it like daily? [22:34:36] so yeah, should be running now theoretically, but I can't check, gotta try to pee on this piwik fire [22:34:54] K, best wishes, thanks for the help [22:34:58] it runs hourly, but I can't remember whether it picks up the missed monthly reports hourly or just daily [22:35:25] and as far as where the metrics show up, they'll be in http://datasets.wikimedia.org/limn-public-data/metrics/multimedia-health/ [22:35:36] so they should overwrite those folders unless there's some permissions issues [22:35:55] the files will be generated on stat1003 and rsynch-ed over to stat1001 (which serves datasets.wikimedia.org) [22:36:52] nuria: btw, I'm trying to figure out how to set up a cron for piwik, in case you want to talk about piwik now [22:37:14] (cron makes it faster so it doesn't re-generate the reports every time someone hits the page) [22:37:31] that's what was killing it (what a terrible default :)) [22:43:28] Analytics-Tech-community-metrics, DevRel-January-2016: NaN values for certain "Time from last patchset" values - https://phabricator.wikimedia.org/T115871#1946028 (Dicortazar) As we can see in http://korma.wmflabs.org/browser/data/json/gerrit.wikimedia.org_mediawiki_extensions_ThrottleOverride-scr-rep-evo... [22:43:45] Analytics-Tech-community-metrics, DevRel-January-2016: NaN values for certain "Time from last patchset" values - https://phabricator.wikimedia.org/T115871#1946029 (Dicortazar) Open>Resolved [22:56:00] Analytics-Kanban: Foundation-only Geowiki stopped updating - https://phabricator.wikimedia.org/T106229#1946061 (Milimetric) Thanks Jaime! I'll move this to done and close it tomorrow if I see geowiki start to catch up. Much appreciated. [22:57:28] Analytics: Look through wikimetrics/scripts and clean them up {dove} - https://phabricator.wikimedia.org/T123956#1946067 (Milimetric) +1 Admin script doesn't just generate data, it schedules reports and does all kinds of cool stuff. It's a permanent part of the code base, and should be used for any scripti... [23:00:38] Analytics, Fundraising-Backlog, Wikimedia-Fundraising: Public dashboards for CentralNotice and Fundraising - https://phabricator.wikimedia.org/T88744#1946085 (AndyRussG) Somewhat related: T122589 [23:33:25] Analytics-EventLogging, MediaWiki-API, Easy, Google-Code-In-2015, Patch-For-Review: ApiJsonSchema implements ApiBase::getCustomPrinter for no good reason - https://phabricator.wikimedia.org/T91454#1946228 (greg) Well, we missed today's branching, but otherwise that is a sane plan. :)