[00:09:47] 10Analytics, 10Product-Analytics: [Spike] Should EventLogging support DNT? - https://phabricator.wikimedia.org/T252438 (10Nuria) This, I think is already a settle request, the new instrumentation client we are working on for Modern Event Platform does not support DNT thus far: https://github.com/wikimedia/medi... [00:14:33] 10Analytics, 10Product-Analytics: [Spike] Should EventLogging support DNT? - https://phabricator.wikimedia.org/T252438 (10Nuria) Ah, no, wait, it DOES support DNT. My bad, cause we discussed this and I thought we decided on not including it. [00:21:08] 10Analytics, 10Product-Analytics: [Spike] Should EventLogging support DNT? - https://phabricator.wikimedia.org/T252438 (10Nuria) >DNT is a failed experiment and has been superseded by ad blockers, browser-based tracking protection, and laws like the GDPR. I can see this argument and do not disagree. >It see... [01:12:35] 10Analytics, 10Product-Analytics: [Spike] Should EventLogging support DNT? - https://phabricator.wikimedia.org/T252438 (10kaldari) >Ah, no, wait, it DOES support DNT. My bad, cause we discussed this and I thought we decided on not including it. @Nuria - So does that mean that the Analytics team has already dec... [03:46:50] 10Analytics: Check home/HDFS leftovers of anomie - https://phabricator.wikimedia.org/T250167 (10tstarling) I don't think we need these files. They are the sort of thing he would have written and immediately executed, not the sort of thing he would have kept coming back to. They are unlikely to be of much value t... [05:08:30] (03PS26) 10Fdans: Add pageview daily dump oozie job to replace Pagecounts-EZ [analytics/refinery] - 10https://gerrit.wikimedia.org/r/595152 (https://phabricator.wikimedia.org/T251777) [05:19:17] (03PS27) 10Fdans: Add pageview daily dump oozie job to replace Pagecounts-EZ [analytics/refinery] - 10https://gerrit.wikimedia.org/r/595152 (https://phabricator.wikimedia.org/T251777) [05:40:37] 10Analytics, 10Operations, 10ops-eqiad: (Need by: TBD) rack/setup/install kafka-jumbo100[789].eqiad.wmnet - https://phabricator.wikimedia.org/T244506 (10Marostegui) @elukey looks like kafka-jumbo1007 is failing to execute any of the NREP commands, while, for instance kafka-jumbo1008 or 1009 are all green. I... [05:46:25] 10Analytics, 10Operations, 10ops-eqiad: (Need by: TBD) rack/setup/install kafka-jumbo100[789].eqiad.wmnet - https://phabricator.wikimedia.org/T244506 (10elukey) Thanks for the ping! Restarted the nagios server on the host and forced a recheck from icinga, let's see if it works. [05:51:29] 10Analytics: Establish if Camus can support TLS encryption + Authentication to Kafka with a minimal code change - https://phabricator.wikimedia.org/T250148 (10elukey) @Ottomata I agree, the task is not super easy :) I do think that we should keep going though, even if it will take 2/3 days of dev time to migrat... [06:01:27] (03PS28) 10Fdans: Add pageview daily dump oozie job to replace Pagecounts-EZ [analytics/refinery] - 10https://gerrit.wikimedia.org/r/595152 (https://phabricator.wikimedia.org/T251777) [06:12:43] 10Analytics, 10Product-Analytics (Kanban): Create Druid tables for Druid datasources in Superset - https://phabricator.wikimedia.org/T251857 (10elukey) [07:09:25] 10Analytics, 10Operations, 10ops-eqiad: (Need by: TBD) rack/setup/install kafka-jumbo100[789].eqiad.wmnet - https://phabricator.wikimedia.org/T244506 (10elukey) Looks good now, removed also the downtime/acks! [07:10:10] 10Analytics, 10Analytics-Kanban, 10Analytics-SWAP, 10Product-Analytics, 10User-Elukey: pip not accessible in new SWAP virtual environments - https://phabricator.wikimedia.org/T247752 (10elukey) [07:10:22] 10Analytics, 10Cassandra, 10User-Elukey: Cassandra3 migration plan proposal - https://phabricator.wikimedia.org/T249756 (10elukey) [07:24:52] 10Analytics, 10Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice, 10Performance-Team (Radar): CentralNotice banners shouldn't be served to bots - https://phabricator.wikimedia.org/T252200 (10Gilles) [07:47:48] joal: so new dynamic partitioning approach: data is pre-loaded partitioned into agent-types and aggregated and then the loop queries each partition and moves the file to the appropriate archive location [08:19:51] (03PS29) 10Fdans: Add pageview daily dump oozie job to replace Pagecounts-EZ [analytics/refinery] - 10https://gerrit.wikimedia.org/r/595152 (https://phabricator.wikimedia.org/T251777) [08:30:23] (03PS30) 10Fdans: Add pageview daily dump oozie job to replace Pagecounts-EZ [analytics/refinery] - 10https://gerrit.wikimedia.org/r/595152 (https://phabricator.wikimedia.org/T251777) [09:15:47] (03PS31) 10Fdans: Add pageview daily dump oozie job to replace Pagecounts-EZ [analytics/refinery] - 10https://gerrit.wikimedia.org/r/595152 (https://phabricator.wikimedia.org/T251777) [10:10:31] (03PS32) 10Fdans: Add pageview daily dump oozie job to replace Pagecounts-EZ [analytics/refinery] - 10https://gerrit.wikimedia.org/r/595152 (https://phabricator.wikimedia.org/T251777) [10:19:50] ha, a test in refinery fails when your OS language is not English [10:19:53] TestLocaleUtil.testGetKnownCountryName:26 expected:<[Ireland]> but was:<[アイルランド]> [10:20:30] Is that "Ireland" In japanese? :D [10:23:58] elukey: yes [10:24:06] Airurando [10:54:24] * elukey lunch! [11:28:02] Hi fdans :) question about the appoach - No mention of time-partitioning - I assume it's done at hour level, at the time as agent-type, with hour-shift as discussed yesterday ? [11:28:27] nice find fdans :) [11:28:31] the test [11:28:43] joal: nono this is about the first job, no hour shifting needed [11:29:07] time partitioning is not needed because it's just hours within a day [11:29:40] fdans: I'm not yet there :/ [11:30:14] joal: can I ask for your help a couple mins in the bc? I'm a little stuck debugging in hive [11:30:20] in hue* [11:30:23] fdans: so first pass to change format (time-partition + hour shift), then split into agent-types partitions and copy files [11:30:38] OMW fdans [12:46:27] (03PS33) 10Fdans: Add pageview daily dump oozie job to replace Pagecounts-EZ [analytics/refinery] - 10https://gerrit.wikimedia.org/r/595152 (https://phabricator.wikimedia.org/T251777) [13:02:14] 10Analytics: Check home/HDFS leftovers of anomie - https://phabricator.wikimedia.org/T250167 (10AMooney) Thanks @tstarling! @elukey and @mforns, go ahead and delete them [13:06:33] 10Analytics: Check home/HDFS leftovers of anomie - https://phabricator.wikimedia.org/T250167 (10elukey) 05Open→03Resolved a:03elukey All cleaned up thanks! [13:26:06] joal: Hi, I was wondering whether you've uploaded a shaded vegas jar to one of the shared servers, which I can reuse? And whether you've posted your kernel.json anywhere I can copy from... [13:26:53] Hi awight_ - I use notebook1003 currently (I should move to stat1007...) [13:27:08] awight_: notebooks def are there as well [13:27:19] awight_: were do you notebook? [13:27:40] joal: I'm currently on stat1005 [13:27:47] ok awight_ [13:28:01] awight_: about vegas, I'm using a modified version of refinery-job [13:28:02] But I'm logged into notebook1003 to peek at your homedir :-) [13:28:06] hehe [13:28:17] /home/joal/refinery-job-0.0.123-SNAPSHOT.jar [13:28:38] awight_: as I was saying, an updated version of refinery-job (to contain more dependencies) [13:29:01] Super-shaded :-) [13:29:09] fatty indeed :) [13:29:37] awight_: /home/joal/.local/share/jupyter/kernels/spark_yarn_scala_libs/kernel.json [13:30:23] ^ no permissions to look at .local [13:30:37] meh [13:31:16] awight_: /home/joal/kernel.json.libs.example [13:31:24] Thanks again! [13:31:27] :) [13:31:42] (03PS34) 10Fdans: Add pageview daily dump oozie job to replace Pagecounts-EZ [analytics/refinery] - 10https://gerrit.wikimedia.org/r/595152 (https://phabricator.wikimedia.org/T251777) [13:34:24] https://github.com/apache/incubator-superset/pull/9789 - let's see if this time it works :D [13:53:30] joal: Awesome jumpstart, I'm already able to run some demo graphs... [13:53:55] \o/ [14:01:46] 10Analytics: Establish if Camus can support TLS encryption + Authentication to Kafka with a minimal code change - https://phabricator.wikimedia.org/T250148 (10Ottomata) I think it might take more than 2/3 days. [14:02:16] 10Analytics: Establish if Camus can support TLS encryption + Authentication to Kafka with a minimal code change - https://phabricator.wikimedia.org/T250148 (10Ottomata) I'd be willing to try Gobblin and/or forked Kafka Connect for some things at this point! [14:09:05] (03PS35) 10Fdans: Add pageview daily dump oozie job to replace Pagecounts-EZ [analytics/refinery] - 10https://gerrit.wikimedia.org/r/595152 (https://phabricator.wikimedia.org/T251777) [14:11:39] 10Analytics, 10I18n, 10RTL: Support right-to-left languages in Wikistats - https://phabricator.wikimedia.org/T251376 (10Aklapper) Should this task be tagged as #Analytics-wikistats or is this about something else? If yes then the project description would welcome a link to the code repository. @Milimetric:... [14:11:42] 10Analytics: Javascript-less Wikistats - https://phabricator.wikimedia.org/T251979 (10Aklapper) Should this task be tagged as #Analytics-wikistats or is this about something else? If yes then the project description would welcome a link to the code repository. @Milimetric: A #good_first_task is a self-contained... [14:13:36] 10Analytics: Establish if Camus can support TLS encryption + Authentication to Kafka with a minimal code change - https://phabricator.wikimedia.org/T250148 (10elukey) @Ottomata I didn't get why it is super difficult to move to form the old consumer config to the newer one, is it a problem for the actual camus co... [14:15:30] (03PS36) 10Fdans: Add pageview daily dump oozie job to replace Pagecounts-EZ [analytics/refinery] - 10https://gerrit.wikimedia.org/r/595152 (https://phabricator.wikimedia.org/T251777) [14:24:33] 10Analytics: Establish if Camus can support TLS encryption + Authentication to Kafka with a minimal code change - https://phabricator.wikimedia.org/T250148 (10Ottomata) I only looked briefly yesterday but from what I gathered: Both old Kafka Consumer and Producer are used. As well as old Message API and Kafka M... [14:28:54] 10Analytics, 10Product-Analytics: [Spike] Should EventLogging support DNT? - https://phabricator.wikimedia.org/T252438 (10Nuria) >So does that mean that the Analytics team has already decided this question in favor of supporting DNT, and thus we should move forwad No, to be honest, I think now is a good time t... [14:31:34] 10Analytics: Use MaxMind DB in piwik geo-location - https://phabricator.wikimedia.org/T213741 (10Aklapper) @Nuria: A #good_first_task is a self-contained, non-controversial task with a clear approach. It should be well-described with pointers to help a completely new contributor. Given the current short task des... [14:33:24] 10Analytics, 10Browser-Support-Opera: Split opera mini in proxy or turbo mode - https://phabricator.wikimedia.org/T138505 (10Aklapper) A #good_first_task is a self-contained, non-controversial task with a clear approach. It should be well-described with pointers to help a completely new contributor. Given the... [14:35:50] 10Analytics, 10good first task: Reportupdater: do not write execution control files in source directories - https://phabricator.wikimedia.org/T173604 (10Aklapper) [14:45:17] it is interesting to see https://docs.cloudera.com/ [14:45:49] the so called "CDP" is a collection of things, from spinning up a (virtual/vm-based) cluster to security etc.. [14:45:58] seems more like what the Hops folks did [14:46:30] the open source distro is called "Cloudera runtime" https://docs.cloudera.com/runtime/7.1.0/index.html [14:46:33] and it doesn't include much [14:46:48] how "open" is CDP I am not sure [14:47:12] I checked some videos and it seems a giant enterprise thing that may break in a million way [14:47:20] super painful without support [14:48:01] I am still convinced that bigtop is the cleanest solution, but it doesn't offer any nice UI of course [15:03:26] 10Analytics, 10Analytics-Kanban, 10Research, 10Patch-For-Review: Proposed adjustment to wmf.wikidata_item_page_link to better handle page moves - https://phabricator.wikimedia.org/T249773 (10Isaac) So a simple improvement is adding a `AND NOT page_is_deleted` to the where clause for the initial `snapshot_p... [15:30:21] 10Analytics, 10Analytics-Kanban: Camus failing to import eqiad.mediawiki.(api|cirrussearch)-request from partitions leaders on kafka-jumbo1006 - https://phabricator.wikimedia.org/T252203 (10Ottomata) [15:30:29] 10Analytics, 10Analytics-Kanban: Camus failing to import eqiad.mediawiki.(api|cirrussearch)-request from partitions leaders on kafka-jumbo1006 - https://phabricator.wikimedia.org/T252203 (10Ottomata) ` 20/05/12 08:58:04 INFO Refine: Successfully refined 142 of 142 dataset partitions into table `event`.`mediawi... [15:33:17] ottomata: soo we have 3 new jumbos! [15:33:44] do you want to add them? [15:34:22] we should also have the new druid nodes [15:35:18] joal: you around? [15:36:30] event data is partitioned by datacenter, and if we make datasets on top of them, I'd like to hide that fact, as I don't think consumers would care at all about what datacenter those come from... was going to brainbounce [15:38:25] elukey: sure can do that! Hm, will try to get to this week, next if not this for sure [15:38:58] milimetric: curious what do you eman make datasets? [15:39:01] mean* [15:39:25] so I'm defining oozie datasets for the events data that we're using in the wikidata job [15:39:34] like, datasets.xml -> mediawiki_page_move_files [15:39:41] ohh oozie right [15:39:57] but the path looks like ${event_data_directory}/mediawiki_page_move/datacenter=${datacenter}/year=${YEAR}/month=${MONTH}/day=${DAY}/hour=${HOUR} [15:40:32] so I don't think that datacenter is very relevant for consumers... was wondering if there's any oozie magic, from the docs it looks like you have to specify everything you can't do like wildcards or anything [15:41:53] Hi milimetric - here I am [15:42:31] hm [15:42:50] are 'consumers' going to use the oozie dataset??? [15:43:48] uh... like, 'consumers' as in us? yeah, I'm using it in this job [15:46:04] milimetric: I have not seen any doc in oozie about wildcards or similar - Shall we test ? [15:46:38] um, I read the docs and they seem pretty clear that everything has to be resolved: https://oozie.apache.org/docs/3.2.0-incubating/CoordinatorFunctionalSpec.html#a6.6.1._coord:currentint_n_EL_Function_for_Synchronous_Datasets [15:47:03] sorry, better link: [15:47:04] https://oozie.apache.org/docs/3.2.0-incubating/CoordinatorFunctionalSpec.html#a5.1._Synchronous_Datasets [15:47:25] "and can be resolved into concrete URIs to identify a particular dataset instance" [15:47:45] "Variables must be resolved at the time a coordinator job is submitted to the coordinator engine" [15:47:58] milimetric: variables, yes - wildcards? [15:48:14] it says it can only include two things: constants and variables [15:48:23] with time constants being the only "close to magic" thing [15:48:48] the other hack I was thinking of is just hard-coding "datacenter=eqiad" in there, but ... I donno... seems problematic too [15:48:50] milimetric: to me time-constants are actually variables :) [15:48:58] milimetric: agreed [15:48:59] yeah, heh [15:49:07] agreed it's problematic, right? [15:49:15] milimetric: yes, problematic [15:49:34] milimetric: if we hard-code, we go for 2 datasets, one perDC, and rely on both [15:49:39] milimetric: what a mess :( [15:50:01] yeah... otherwise consumers have to specify both and hardcode the datacenters in their properties file, no? [15:50:02] milimetric: particularly with some folder being empty every now and then [15:50:41] oh, joal, how about a simple oozie job to unify? [15:50:50] so we have _SYNCED or something as a new flag [15:51:10] when both (all to be future proof) datacenters are available, we write _SYNCED and then consumers can use that? [15:51:12] milimetric: I'd still like to test the wildcard (or more precisely datacenter=[eqiad,codfw]) [15:51:35] milimetric: doable [15:52:25] milimetric: we would pick a datacenter folder and write a different done flag - that could work - but doesn't simplifies :S [15:53:35] trouble is, codfw doesn't usually have data [15:53:47] so if you depend on it, your jobs will wait forever [15:53:54] indeed ottomata, I mentioned that above :( [15:53:57] phj [15:53:58] oh [15:53:59] :) [15:54:27] And when we do failover (either for test or real), no data in eqiad [15:54:33] Meh [15:54:48] * joal really hopes wildcards could work :( [15:55:20] the canary events task might help with that, but it could also hurt other alerts, e.g. if only the canary event was produced but all real producers were faliing [15:56:02] joal: but if the wildcards work, wouldn't it just wait until both are available? [15:56:12] 10Analytics, 10Analytics-Kanban, 10Research, 10Patch-For-Review: Proposed adjustment to wmf.wikidata_item_page_link to better handle page moves - https://phabricator.wikimedia.org/T249773 (10Milimetric) >>! In T249773#6129599, @Isaac wrote: > I am starting to think though that we essentially have to create... [15:56:34] I'm going to the batcave [15:56:41] milimetric: ah indeed! Meh - [15:56:48] joining milimetric [16:22:33] 10Analytics: [Spike] Explore goblin as an alternative to camus - https://phabricator.wikimedia.org/T252560 (10Nuria) [16:27:44] 10Analytics: [Spike] Explore goblin as an alternative to camus - https://phabricator.wikimedia.org/T252560 (10Nuria) An idea: - we can use goblin and replicate consumption of 1 stream (1 off) and let's evaluate how hard that is [16:53:14] 10Analytics: Update sqoop before labs views change - https://phabricator.wikimedia.org/T252565 (10Milimetric) [16:54:34] 10Analytics: Update sqoop before labs views change - https://phabricator.wikimedia.org/T252565 (10Milimetric) [17:08:24] 10Analytics, 10Discovery-Analysis, 10Product-Analytics: Wikimedia Discovery Golden timer on stat1007 keeps logging errors - https://phabricator.wikimedia.org/T252365 (10LGoto) p:05Triage→03Medium [17:08:44] 10Analytics, 10Discovery-Analysis, 10Product-Analytics: Wikimedia Discovery Golden timer on stat1007 keeps logging errors - https://phabricator.wikimedia.org/T252365 (10mpopov) a:03mpopov [17:28:33] * elukey off! [18:06:18] 10Analytics, 10Analytics-Kanban, 10Research, 10Patch-For-Review: Proposed adjustment to wmf.wikidata_item_page_link to better handle page moves - https://phabricator.wikimedia.org/T249773 (10Isaac) Yeah, I can live with the duplicates until the more long-term fix for missing data is made. There are three w... [18:18:48] hey milimetric - is now a good time? [18:23:23] milimetric: did you guys resolved the dc prefix issue cause it seems that use that data for any purpose we need to think about that , seems that (if we are not considering 'active' /'active' setup) the dc needs to be harcoded to an env variable for 'active dc' like -DactiveDC = $active_DC that might be set via puppet [18:23:49] nuria: not solved, we are talking about it [18:23:51] milimetric/joal: maybe i am missing why this would not work [18:23:59] nuria: and no easy soution :( [18:24:09] joal: in bc? [18:24:31] waiting for milimetric now, but let's batcave and talk waiting for im if you have a moment [18:25:15] joal: yes [18:44:04] ottomata: heya :) [18:44:13] hiya [18:44:19] ottomata: would ou have a minute for a quick talk in da cave with nuria ? [18:44:21] sure! [19:37:55] 10Analytics: Refine event pipeline at this time refines data in hourly partitions without knowing if the partition is complete - https://phabricator.wikimedia.org/T252585 (10Nuria) [19:39:49] 10Analytics: Refine event pipeline at this time refines data in hourly partitions without knowing if the partition is complete - https://phabricator.wikimedia.org/T252585 (10Nuria) ping @Milimetric , this ticket documents our recent discussion in batcave about issues brought up on the events recent job. (cc @JA... [19:59:54] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Event-Platform, and 2 others: Vertical: Migrate SearchSatisfaction EventLogging event stream to Event Platform - https://phabricator.wikimedia.org/T249261 (10Ottomata) t(ಠ益ಠt) PHP! From https://www.php.net/manual/en/language.operators.array.php... [20:00:12] 10Analytics, 10Analytics-EventLogging, 10Product-Analytics: EditAttemptStep sent event with "ready_timing": -18446744073709543000 - https://phabricator.wikimedia.org/T251772 (10mpopov) 05Open→03Resolved a:03mpopov Thanks for your input, folks! I've split the outcomes into: - T252593: making the clien... [20:04:38] (03PS1) 10QChris: Allow “Gerrit Managers” to import history [analytics/wmde/TW/edit-conflicts] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/596051 [20:04:42] (03CR) 10QChris: [V: 03+2 C: 03+2] Allow “Gerrit Managers” to import history [analytics/wmde/TW/edit-conflicts] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/596051 (owner: 10QChris) [20:04:50] (03PS1) 10QChris: Import done. Revoke import grants [analytics/wmde/TW/edit-conflicts] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/596052 [20:04:54] (03CR) 10QChris: [V: 03+2 C: 03+2] Import done. Revoke import grants [analytics/wmde/TW/edit-conflicts] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/596052 (owner: 10QChris) [20:09:30] 10Analytics, 10Discovery-Analysis, 10Product-Analytics: Wikimedia Discovery Golden timer on stat1007 keeps logging errors - https://phabricator.wikimedia.org/T252365 (10mpopov) Looks like Reportupdater is trying to backfill tile request data from 2020-01-02 but the scripts are returning no data because there... [20:11:37] 10Analytics, 10Discovery-Analysis, 10Product-Analytics: Decomission maps metrics module from wikimedia/discovery/golden data retrieval - https://phabricator.wikimedia.org/T252365 (10mpopov) [20:12:24] 10Analytics, 10Discovery-Analysis, 10Product-Analytics (Kanban): Decomission maps metrics module from wikimedia/discovery/golden data retrieval - https://phabricator.wikimedia.org/T252365 (10mpopov) [20:47:15] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Event-Platform, and 2 others: Decommission EventLogging backend components by migrating to MEP - https://phabricator.wikimedia.org/T238230 (10Ottomata) And `mw.eventLog.logEvent("Test", {"OtherMessage": "test"}` works from test.wikipedia.org too!... [20:52:49] nuria: I missed the earlier pings, I didn't get them on my phone for some reason, but did you and joal talk and do you want to sync up now? [20:53:03] I'm working again later tonight if that's better [20:54:43] milimetric: we did talk, some thoughts are here: https://phabricator.wikimedia.org/T252585 [21:01:03] 10Analytics: Refine event pipeline at this time refines data in hourly partitions without knowing if the partition is complete - https://phabricator.wikimedia.org/T252585 (10Milimetric) Ok, so I don't see any reason to block work on the wikidata item_page_link improvement until we have this working perfectly.... [21:01:30] k, I replied and I'll submit my job with that assumption in mind, using datacenter=all [21:08:27] 10Analytics: Refine event pipeline at this time refines data in hourly partitions without knowing if the partition is complete - https://phabricator.wikimedia.org/T252585 (10Nuria) >Another solution is to run a delayed job for each event dataset that we need. This would check a folder that has datacenter=eqiad...