[00:18:01] (03PS1) 10GoranSMilovanovic: Initial [analytics/wmde/WiktionaryCognateDashboard] - 10https://gerrit.wikimedia.org/r/452589 [00:18:50] (03CR) 10GoranSMilovanovic: [V: 032 C: 032] Initial [analytics/wmde/WiktionaryCognateDashboard] - 10https://gerrit.wikimedia.org/r/452589 (owner: 10GoranSMilovanovic) [00:31:11] (03PS1) 10GoranSMilovanovic: README [analytics/wmde/WiktionaryCognateDashboard] - 10https://gerrit.wikimedia.org/r/452590 [00:31:38] (03CR) 10GoranSMilovanovic: [V: 032 C: 032] README [analytics/wmde/WiktionaryCognateDashboard] - 10https://gerrit.wikimedia.org/r/452590 (owner: 10GoranSMilovanovic) [00:37:07] (03PS1) 10GoranSMilovanovic: GitHub link [analytics/wmde/WiktionaryCognateDashboard] - 10https://gerrit.wikimedia.org/r/452591 [00:37:29] (03CR) 10GoranSMilovanovic: [V: 032 C: 032] GitHub link [analytics/wmde/WiktionaryCognateDashboard] - 10https://gerrit.wikimedia.org/r/452591 (owner: 10GoranSMilovanovic) [00:42:19] Wiktionary Cognate Dashboard completed and running from http://wdcm.wmflabs.org/Wiktionary_CognateDashboard/ - Go play. [12:02:39] helooooo :] [12:32:04] moorning! [12:32:53] heeeey fdans :] [12:33:47] mforns: yesterday I finished applying the change you suggested in the join :) should be good for a new review [12:33:56] fdans, ok, will look [12:34:03] graciaaaas [13:53:55] (03PS1) 10Mforns: Allow for null structs in WhitelistSanitization.scala [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/452673 (https://phabricator.wikimedia.org/T199902) [14:29:37] (03CR) 10Ottomata: [C: 031] "AH nice find!" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/452673 (https://phabricator.wikimedia.org/T199902) (owner: 10Mforns) [14:35:43] thx! [14:36:36] (03CR) 10Mforns: [C: 032] Allow for null structs in WhitelistSanitization.scala [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/452673 (https://phabricator.wikimedia.org/T199902) (owner: 10Mforns) [14:36:52] (03CR) 10Mforns: [V: 032 C: 032] Allow for null structs in WhitelistSanitization.scala [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/452673 (https://phabricator.wikimedia.org/T199902) (owner: 10Mforns) [14:39:24] !log Starting deployment of refinery-source using jenkins [14:39:26] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:41:04] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, 10Services (watching): Modern Event Platform: Scalable Event Intake - https://phabricator.wikimedia.org/T201068 (10Ottomata) > As an engineer, I want to be able to guarantee production of the event and be able to retry until the event... [14:41:59] (03PS1) 10Mforns: Update changelog.md for version v0.0.70 [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/452681 [14:42:24] (03CR) 10Mforns: [V: 032 C: 032] Update changelog.md for version v0.0.70 [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/452681 (owner: 10Mforns) [14:53:01] ottomata: https://github.com/wikimedia/mediawiki-extensions-EventLogging/blob/ea95faf8369148911898a8414dd1fc3a8ea2bbb3/includes/EventLogging.php#L36 says we POST? [14:53:40] but this says we concat the event on the end: https://github.com/wikimedia/mediawiki-extensions-EventLogging/blob/ea95faf8369148911898a8414dd1fc3a8ea2bbb3/includes/EventLogging.php#L75 [14:54:01] and then POST it?! https://github.com/wikimedia/mediawiki-extensions-EventLogging/blob/ea95faf8369148911898a8414dd1fc3a8ea2bbb3/includes/EventLogging.php#L29 [14:54:03] milimetric: POST or not POST, the data is in the URL [14:54:04] https://github.com/wikimedia/mediawiki-extensions-EventLogging/blob/ea95faf8369148911898a8414dd1fc3a8ea2bbb3/includes/EventLogging.php#L75-L76 [14:54:12] yeah, exactly [14:54:15] yeah [14:54:20] that is weird [14:54:23] right [14:54:27] varnish probably is accepting both [14:54:41] but, varnish has no way to get the data out of the POST body [14:54:52] it seems like maybe varnish could just read the post data but maybe that's hard to do or something? [14:55:02] yeah, i think it can't, it isn't in the varnish shared log [14:55:05] i think...checking [14:55:19] oh, ok, so that'd be a serious limitation [14:55:22] naw, you can't [14:55:36] https://serverfault.com/questions/734467/varnishlog-accessing-post-body-varnish-4-x [14:55:41] Having said that, it is possible to https://varnish-cache.org/docs/trunk/users-guide/vcl-inline-c.html code into your VCL. Therefore, technically you could access the POST data using some custom C code and use https://varnish-cache.org/docs/4.0/reference/vmod_std.generated.html#func-log to write the bits you care about to the logs. [14:55:46] probably not really the best solution either way [15:01:45] standup? [15:02:04] OHHH [15:02:08] coming [15:02:23] milimetric: ^ [15:02:34] nuria_: maybe? [15:09:20] 10Analytics: How to get display statistics of the content publised on Commons - https://phabricator.wikimedia.org/T201180 (10Milimetric) No @WMDE-leszek we have a lot of other stuff we need to do first. It is a relatively small task, so I'd be happy to mentor someone doing it. I'll also try to grab it when I'm... [15:16:02] 10Analytics, 10MinervaNeue, 10Readers-Web-Backlog, 10Design: [Spike 8hrs] Sticky header instrumentation - https://phabricator.wikimedia.org/T199157 (10Jdlrobson) Since there is no "points 0" on this task it means we haven't talked about it. We did an async estimation and estimations are all over the place.... [15:35:33] ottomata, now I'm here :) [15:35:38] ottomata, when you're done with meeting, can you help me with refinery-source deployment? [15:44:21] mforns: ya sure give ma few [15:44:28] dsaez: uhhh, can't remember what! :) [15:44:29] oh yeah! [15:44:36] we made some 'large' spark kernels [15:44:42] they should work for your stuff i think [15:46:19] see https://wikitech.wikimedia.org/wiki/SWAP#Spark and also https://wikitech.wikimedia.org/wiki/SWAP#Custom_Spark_Kernels [15:47:31] oh, let me see [15:51:46] !log removed /srv/geowiki from stat1006 [15:51:48] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [15:51:54] yeaaah! [15:52:11] :* [15:52:31] that's lovely! [15:54:45] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: pyspark2 job killed by YARN for exceeding memory limits - https://phabricator.wikimedia.org/T201519 (10diego) My mistake, I was not using the "large" option. This work smoothly! Thank you @Ottomata and @JAllemandou your amazing! [15:55:00] thank you Ottomata and joal! this works smoothly!!! [15:55:09] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Turn off old geowiki jobs - https://phabricator.wikimedia.org/T190059 (10fdans) @Nuria @Milimetric we've disabled the jobs and removed /srv/geowiki from stat1006. Do we want to also remove the geowiki site from thorium? If not this is done. [15:56:09] very cool! [16:03:35] ottomata, first jenkins run of https://integration.wikimedia.org/ci/job/analytics-refinery-release/m2release/ had the error I commented in standup (connection error, not related to code) second run of same job finished packaging but errored on maven-release-plugin because the tag v0.0.70 did already exist [16:03:46] should I start over with v0.0.71? [16:09:15] ergh [16:09:24] mforns: won't hurt i suppose? [16:09:30] or i can delete the 0.0.70 tag? [16:09:38] oh, yes please [16:09:42] k let's try that... [16:10:12] done [16:10:16] thx@ [16:11:22] ottomata, wait I'm a bit confused... [16:11:32] can we look at this together for a sec in the bc? [16:16:33] I think somehow the v0.0.70 tag is still there, when I open https://integration.wikimedia.org/ci/job/analytics-refinery-release/m2release/ it defaults to 0.0.71 (0.0.72-SNAPSHOT) [16:16:54] seems like grabbing 0.0.70 from archiva? and adding 1? [16:17:29] 10Analytics, 10MinervaNeue, 10Readers-Web-Backlog, 10Design: [Spike ??hrs] Sticky header instrumentation - https://phabricator.wikimedia.org/T199157 (10Jdlrobson) [16:18:16] 10Analytics, 10Discovery-Search (Current work), 10Patch-For-Review: Use kafka for communication from analytics cluster to elasticsearch - https://phabricator.wikimedia.org/T198490 (10EBernhardson) This is all deployed and ran a limited push from analytics to prod last weekend. This next weekend it will do th... [16:28:25] OH, mforns hmm, i deleted the 0.0.70 tag from gerrit [16:28:29] i didn't delete any artifacts from archiva [16:28:39] maybe we need to do that too [16:28:42] ok [16:28:42] mforns: going to bc [16:28:44] ok [16:32:28] 10Analytics, 10Analytics-Kanban, 10User-Elukey: Q1 2018/19 Analytics procurement - https://phabricator.wikimedia.org/T198694 (10Cmjohnson) [16:42:56] 10Analytics, 10Operations: rack/setup/install 2 new hadoop master/standby systems in eqiad - https://phabricator.wikimedia.org/T201939 (10RobH) p:05Triage>03Normal [16:46:08] 10Analytics, 10Operations: rack/setup/install 2 new hadoop master/standby systems in eqiad - https://phabricator.wikimedia.org/T201939 (10RobH) Assigned to @elukey for hostname feedback, but as they are on vacation perhaps someone else in #analytics would be able to provide feedback on hostname? [16:46:20] 10Analytics, 10Operations: rack/setup/install 2 new hadoop master/standby systems in eqiad - https://phabricator.wikimedia.org/T201939 (10RobH) a:05elukey>03Ottomata [16:56:54] 10Analytics, 10Operations: rack/setup/install 2 new hadoop master/standby systems in eqiad - https://phabricator.wikimedia.org/T201939 (10Ottomata) Hm, tough question! I'd be ok with analytics-master1001 and analytics-master1002. Let's do it! [16:58:33] 10Analytics, 10Operations: rack/setup/install analytics-master100[12].eqiad.wmnet - https://phabricator.wikimedia.org/T201939 (10RobH) a:05Ottomata>03Cmjohnson [17:23:10] Hi, Analytics! I'm trying to get the list of referrers for one specific page for a 24 hour window. (Specifically, for https://it.wikipedia.org/wiki/Wikipedia:Comunicato_3_luglio_2018). I know that Hive has this data in wmf.webrequest table, but I don't want to accidentally break something (or more to the point, page someone). Is there anything in particular that I should do in order to avoid excessive load? [17:23:28] ottomata: guessing you might have an opinion/suggestion here, but the question is very much open to anyone. [17:27:19] Actual query would be something like `SELECT uri_host, uri_path, referer FROM webrequest WHERE (uri_host='it.wikipedia.org' or uri_host='it.m.wikipedia.org') AND (uri_path='/wiki/Wikipedia:Comunicato_3_luglio_2018' OR uri_path='/w/Wikipedia:Comunicato_3_luglio_2018') AND year=2018 AND month=8 AND day=13` [17:29:01] !log Finished deployment of refinery-source using jenkins [17:29:03] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [17:32:14] marlier: that should be fine, I'd also add webrequest_source partition to your where clause [17:32:16] to limit the data more [17:32:26] you probably just want webrequest_source='text' [17:32:29] Sure, will do. [17:35:30] ottomata: might be nice to have the valid values for webrequest_source documented on https://wikitech.wikimedia.org/wiki/Analytics/Data_Lake/Traffic/Webrequest [17:36:01] marlier: aye, it corresponds to the varnish cache clusters [17:38:18] ottomata, yes, https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/452674/ is OK to merge, thanks! [17:47:41] 10Analytics, 10Operations, 10ops-eqiad: rack/setup/install analytics-master100[12].eqiad.wmnet - https://phabricator.wikimedia.org/T201939 (10Cmjohnson) [17:47:58] 10Analytics, 10Operations, 10ops-eqiad: rack/setup/install analytics-master100[12].eqiad.wmnet - https://phabricator.wikimedia.org/T201939 (10Cmjohnson) [17:48:31] (03PS1) 10Mforns: Fix typo in python/utils.py [analytics/refinery] - 10https://gerrit.wikimedia.org/r/452719 [17:49:09] 10Analytics, 10Operations, 10ops-eqiad: rack/setup/install analytics-master100[12].eqiad.wmnet - https://phabricator.wikimedia.org/T201939 (10Cmjohnson) @ottomata the name is entirely too long for labels and tracking. can we shorten it a bit? [17:50:12] fdans, can you quickly review this? it's a typo that sneaked in the last merge.. I'm about to deploy refinery: https://gerrit.wikimedia.org/r/#/c/analytics/refinery/+/452719/ [17:53:06] mforns: looking [17:54:34] (03CR) 10Fdans: [C: 032] Fix typo in python/utils.py [analytics/refinery] - 10https://gerrit.wikimedia.org/r/452719 (owner: 10Mforns) [17:54:46] fdans, thank youuuu :] [17:54:58] fdans, I don't think this code is deployed yet [17:55:32] at least, I can not see the corresponding changes in stat1004.eqiad.wmnet:/mnt/hdfs/wmf/refinery/current [17:56:01] (03CR) 10Ottomata: [V: 032 C: 032] Fix typo in python/utils.py [analytics/refinery] - 10https://gerrit.wikimedia.org/r/452719 (owner: 10Mforns) [18:03:32] 10Analytics, 10Operations, 10ops-eqiad: rack/setup/install analytics-master100[12].eqiad.wmnet - https://phabricator.wikimedia.org/T201939 (10RobH) >>! In T201939#4502422, @Cmjohnson wrote: > @ottomata the name is entirely too long for labels and tracking. can we shorten it a bit? This was discussed in IRC... [18:03:55] !log Starting deployment of refinery using scap and refinery-deploy-to-hdfs [18:03:56] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [18:29:03] milimetric: yt? [18:29:37] hey [18:30:35] ottomata: what's up [18:31:01] Pchelolo: actually answer here since i just started pinging milimetric too [18:31:29] milimetric: am working on the rfc for scalable event intake [18:31:38] ah cool [18:31:42] and wanted to brain bounce opinion [18:31:46] what do you think? [18:31:55] should we write something new? or just keep using and augmenting eventbus? [18:31:56] yeah, let's bounce them brains, that's what they're for [18:32:03] ottomata: ye [18:32:26] it's all about performance, this is the single point of intake, right? [18:32:31] *going to be [18:32:58] EventBus did have some issues with timeouts that were never solved, also I thought the plan was to completely merge eventlogging and eventbus [18:33:13] hmm, no its not all about performance [18:33:20] the scalable part is about horizontal scalability [18:33:35] Pchelolo: errrrrr yes, but not from a tech point of view [18:33:38] from a usage point of view [18:33:40] I include that in performance, in the sense that it has to be able to handle the full firehose [18:34:18] hm ok. yeah, there are still some unexplained timeouts in eventbus for sure [18:34:25] i also don't love the tornado async code [18:34:39] it isn't as clear to me or as easy to work with as say nodejs promises [18:34:44] that includes scaling, so if we can scale EventBus to handle what we need, and there are no other issues like maintainability and ops not wanting a third major technology (Python) implementing big pieces of the stack [18:35:06] those are issues for sure [18:35:10] if it was me, I'd rewrite it in Node [18:37:48] define some performance requirements, and implement the same ideas that are in EventBus but with a more natural environment (like you said better ways to do async). [18:38:09] Pchelolo: have you all considered TypeScript for our node work, btw [18:38:24] milimetric: no, didn't look into that [18:38:35] milimetric: performance requirements are really only that it is horizontally scalable, whatever python or node can do will be sufficient [18:38:44] not for the lack of interest, for the lack of time [18:39:27] ok, yeah, it seems worth a quick look here if this is going to be a major piece of infrastructure. There's supposedly an easy migration path from plain JS and it gives you typing which could make maintenance a little less stressful for folks not used to JS [18:39:45] in node milimetric? [18:39:51] yea [18:39:58] typescript just puts a type system on top of JS [18:40:19] and ottomata what about Kafka clients, are those of equal quality in js/python or not? [18:40:23] hm, yeah could be cool, not sure i'd want to introduce that for this [18:40:30] i'd prefer to use something like mediawiki-service-template [18:40:46] yeah, definitely, it wouldn't clash with that, you can use it for just part of your code if you want [18:40:50] milimetric: yes/no? the client we'd use for node is built on librdkafka [18:41:13] the kafka client in node is of a decent quality and fairly actively maintaned [18:41:34] ya, change-prop uses it for producing, and we use it for consuming in eventstreams [18:41:44] Pchelolo: qq only sort of related [18:42:02] in Java, I can read URLs like file:/// and http:// (and even hdfs:// if configured) with the same function [18:42:06] can I do somethign like that in node? [18:42:22] googling but not sure [18:42:35] don't think it's in core node, but there should be a package [18:42:42] there's a package for everything [18:43:11] haha [18:43:13] true [18:43:16] yeah, core just has like readFs and only basic http stuff [18:44:01] (btw on migrating to TypeScript one file at a time: https://dev.to/suhas_chatekar/converting-a-javascript-project-to-typescript-one-file-at-a-time) [18:44:48] !log Finished deployment of refinery using scap and refinery-deploy-to-hdfs [18:44:50] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [18:46:03] so, given limited knowledge it sounds like you both would lean towards rewriting? [18:55:16] (03CR) 10Mforns: [C: 032] Change appInstallID labels to hash in EL sanitization whitelist [analytics/refinery] - 10https://gerrit.wikimedia.org/r/450588 (https://phabricator.wikimedia.org/T199902) (owner: 10Mforns) [18:55:33] (03PS2) 10Mforns: Change appInstallID labels to hash in EL sanitization whitelist [analytics/refinery] - 10https://gerrit.wikimedia.org/r/450588 (https://phabricator.wikimedia.org/T199902) [18:55:38] (03CR) 10Mforns: [V: 032 C: 032] Change appInstallID labels to hash in EL sanitization whitelist [analytics/refinery] - 10https://gerrit.wikimedia.org/r/450588 (https://phabricator.wikimedia.org/T199902) (owner: 10Mforns) [18:57:00] 10Analytics, 10Analytics-Kanban, 10Datasets-General-or-Unknown, 10Documentation: Missing documentation for pageviews dataset - https://phabricator.wikimedia.org/T201653 (10Milimetric) Something really really strange is happening. I added this file back in January: https://github.com/wikimedia/puppet/blob/... [18:57:26] !log Starting deployment of refinery using scap and refinery-deploy-to-hdfs [18:57:40] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [19:35:20] !log Finished deployment of refinery using scap and refinery-deploy-to-hdfs [19:35:22] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [19:38:03] Pchelolo: another q for you...do you thikn we need an RFC for the scalable event intake component? i'd mainly use it to decide on the q I asked above [19:38:11] to keep eventbus or to do something else [20:27:07] 10Analytics, 10Analytics-EventLogging, 10MediaWiki-extensions-WikimediaEvents, 10Page-Issue-Warnings, and 6 others: Provide standard/reproducible way to access a PageToken - https://phabricator.wikimedia.org/T201124 (10Niedzielski) @Tbayer and I discussed and reviewed. Here are our conclusions: - For pa... [20:56:06] 10Analytics, 10EventBus, 10Product-Analytics, 10MW-1.32-release-notes (WMF-deploy-2018-08-21 (1.32.0-wmf.18)), and 2 others: Load change tags into the Analytics Data Lake on a daily basis - https://phabricator.wikimedia.org/T201062 (10Jdforrester-WMF) [21:07:49] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, 10Services (watching): [WIP] RFC: Modern Event Platform: Scalable Event Intake Service - https://phabricator.wikimedia.org/T201963 (10Ottomata) p:05Triage>03Normal [22:11:42] hey, can you remind me where ican lookup browser share stats? I'm keen to see what % of our traffic opera mini is these days but can't find it on https://stats.wikimedia.org/v2/#/all-projects and https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/BrowserReports appears to be out of date? [22:21:44] looks like https://analytics.wikimedia.org/dashboards/browsers/#mobile-site-by-browser/browser-family-and-major-tabular-view and https://turnilo.wikimedia.org/ was what i was looking for [22:50:01] 10Analytics, 10Analytics-EventLogging, 10MediaWiki-extensions-WikimediaEvents, 10Page-Issue-Warnings, and 6 others: Provide standard/reproducible way to access a PageToken - https://phabricator.wikimedia.org/T201124 (10Tbayer) >>! In T201124#4502862, @Niedzielski wrote: > @Tbayer and I discussed and review... [22:53:13] (03PS1) 10Fdans: Revert "Adds empty dir removal to hive partition dropping jobs" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/452861 [22:55:21] marlier, ottomata: it's also a good idea to nice the request in case it could generate too much load https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hive/Queries#Run_long_queries_in_a_screen_session_and_in_the_nice_queue [22:59:10] (03PS2) 10Fdans: Revert "Adds empty dir removal to hive partition dropping jobs" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/452861 [23:02:26] fdans: i merge ^? [23:02:43] ottomata: one sec, I'll merge, solving a conflict [23:02:46] k [23:02:51] oh you can merge that's not pupppet [23:02:53] (03PS3) 10Fdans: Revert "Adds empty dir removal to hive partition dropping jobs" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/452861 [23:02:54] right ok you go for it :) [23:03:38] ottomata: yeah spotted a typo, will deploy revert and then send a new change [23:04:21] just making sure mforns 's change stays deployed [23:04:58] (03CR) 10Fdans: [V: 032 C: 032] Revert "Adds empty dir removal to hive partition dropping jobs" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/452861 (owner: 10Fdans) [23:07:34] !log starting deployment of refinery via scap [23:07:36] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log