[00:11:43] <wikibugs>	 10Analytics, 10Developer-Advocacy, 10Product-Analytics, 10Documentation: Develop EventLogging schema for documentation feedback gadget - https://phabricator.wikimedia.org/T211638 (10srishakatux) @fdans Thanks! We don't need to keep the data beyond 90-day window for now. Does the schema look good to you and...
[00:23:26] <mforns>	 byeeeeeee!
[00:47:37] <wikibugs>	 (03CR) 10Nuria: [C: 03+2] Add tooltip Vue directive and use it [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/478824 (https://phabricator.wikimedia.org/T177950) (owner: 10Milimetric)
[00:50:24] <wikibugs>	 (03CR) 10Milimetric: "Thanks for the check, I played with the prod build too, making the tooltip file huge, and it seemed to still affect rendering, but not fir" [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/478824 (https://phabricator.wikimedia.org/T177950) (owner: 10Milimetric)
[00:51:18] <wikibugs>	 (03CR) 10Milimetric: [C: 04-1] "Still need to fill out the tooltips all over the site." [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/478824 (https://phabricator.wikimedia.org/T177950) (owner: 10Milimetric)
[01:08:22] <wikibugs>	 10Analytics, 10Anti-Harassment, 10Product-Analytics: Add partial blocks to mediawiki history tables - https://phabricator.wikimedia.org/T211950 (10nettrom_WMF)
[05:16:53] <wikibugs>	 10Analytics, 10Product-Analytics: Investigate referrer class change on Chrome Mobile from September 13, 2018 - https://phabricator.wikimedia.org/T211077 (10Nuria) Closing ticket, added note to : https://wikitech.wikimedia.org/wiki/Analytics/Data_Lake/Traffic/Webrequest#Changes_and_known_problems_since_2015-03-04
[05:16:54] <wikibugs>	 10Analytics, 10Product-Analytics: Investigate referrer class change on Chrome Mobile from September 13, 2018 - https://phabricator.wikimedia.org/T211077 (10Nuria) 05Open→03Resolved
[05:22:50] <wikibugs>	 10Analytics, 10Product-Analytics: Event counts from Mysql and Hive don't match. Refine is persisting data from crawlers. - https://phabricator.wikimedia.org/T210006 (10Nuria) p:05Normal→03High
[05:28:10] <wikibugs>	 10Analytics, 10Growth-Team, 10Product-Analytics, 10Patch-For-Review: Add EditAttemptStep properties to the schema whitelist - https://phabricator.wikimedia.org/T208332 (10Nuria) 05Open→03Resolved
[05:29:52] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10Product-Analytics: [BUG] User agent parsing error for MobileWikiAppSearch table - https://phabricator.wikimedia.org/T211833 (10Nuria)
[05:44:10] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Final steps to expose project family unique devices data - https://phabricator.wikimedia.org/T167539 (10Nuria) Docs to be updated located here: https://wikitech.wikimedia.org/wiki/Analytics/AQS/Unique_Devices
[05:59:22] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10Product-Analytics: [BUG] User agent parsing error for MobileWikiAppSearch table - https://phabricator.wikimedia.org/T211833 (10Nuria) uuid="8522d275ebcb5f7cbd82f44ca27c6dcf"  Raw data seems correct   {"dt": "2018-11-30T10:00:00Z", "event": {"action": "results", "app_install...
[07:45:31] <fdans>	 helloooo elukey! https://github.com/wikimedia/incubator-superset/pull/1
[07:47:01] <wikibugs>	 10Analytics, 10Developer-Advocacy, 10Product-Analytics, 10Documentation: Develop EventLogging schema for documentation feedback gadget - https://phabricator.wikimedia.org/T211638 (10fdans) @srishakatux we reviewed the schema during our tasking meeting and it looks good! Feel free to put it in metawiki.
[07:48:05] <elukey>	 fdans: looks good! Le
[07:48:24] <elukey>	 if Andrew is around later on let's merge it, otherwise on monday
[07:48:31] <fdans>	 yesss
[07:48:44] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10Product-Analytics: Bug: can't make a YoY time series chart in Superset - https://phabricator.wikimedia.org/T210687 (10fdans) I've added a PR in our github fork that fixes this particular problem:  https://github.com/wikimedia/incubator-superset/pull/1
[08:04:19] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats, 10Patch-For-Review: Add a tooltip to all non-obvious concepts like split categories, abbreviations - https://phabricator.wikimedia.org/T177950 (10fdans) Current look of the tooltip in my browser:  {F27549355}
[08:14:59] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Final steps to expose project family unique devices data - https://phabricator.wikimedia.org/T167539 (10fdans) Examples updated in that page. Also took a look at: - https://wikitech.wikimedia.org/wiki/Analytics/Data_Lake/Traffic/Unique_Devices - https://me...
[08:16:17] <wikibugs>	 (03CR) 10Fdans: "Looking good! Just a tiny comment about imports and making sure the style is correct." (033 comments) [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/478824 (https://phabricator.wikimedia.org/T177950) (owner: 10Milimetric)
[08:16:28] <wikibugs>	 (03CR) 10Fdans: [C: 03+2] Add tooltip Vue directive and use it [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/478824 (https://phabricator.wikimedia.org/T177950) (owner: 10Milimetric)
[08:16:39] <wikibugs>	 (03CR) 10Fdans: Add tooltip Vue directive and use it [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/478824 (https://phabricator.wikimedia.org/T177950) (owner: 10Milimetric)
[08:17:28] <wikibugs>	 (03CR) 10Fdans: "Sorry for the accidental +2. I meant to comment that I don't think you have to fill the tooltips in this change, maybe send a separate one" [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/478824 (https://phabricator.wikimedia.org/T177950) (owner: 10Milimetric)
[08:34:41] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10DBA, 10Data-Services, and 3 others: Create materialized views on Wiki Replica hosts for better query performance - https://phabricator.wikimedia.org/T210693 (10Banyek) We didn't talked about this so far, but these views doesn't ask for having proper indexes?
[11:52:07] * elukey lunch!
[12:24:36] <wikibugs>	 (03CR) 10Lucas Werkmeister (WMDE): [C: 03+1] "holding off on the merge because Tarrow also wanted to review, I believe" [analytics/wmde/toolkit-analyzer] - 10https://gerrit.wikimedia.org/r/475807 (https://phabricator.wikimedia.org/T209399) (owner: 10Michael Große)
[14:01:13] <wikibugs>	 (03CR) 10Tarrow: [C: 03+2] Update metric's items and properties automatically [analytics/wmde/toolkit-analyzer] - 10https://gerrit.wikimedia.org/r/475807 (https://phabricator.wikimedia.org/T209399) (owner: 10Michael Große)
[14:01:24] <wikibugs>	 (03Merged) 10jenkins-bot: Update metric's items and properties automatically [analytics/wmde/toolkit-analyzer] - 10https://gerrit.wikimedia.org/r/475807 (https://phabricator.wikimedia.org/T209399) (owner: 10Michael Große)
[14:24:42] <joal>	 Wow, got IRC issues - lost backlog - hopefully I have not been pinged?
[14:25:42] <elukey>	 joal: you have not been pinged! (until now :P)
[14:25:48] <joal>	 ottomata: Heya - I have tested presto/hive, but I have permission issues for hive (/etc/hive/conf/hive-site.xml), and presto doesn't manage manage to connect to metastore
[14:25:57] <joal>	 Thanks elukey - ping now works :)
[14:26:11] <ottomata>	 hmmmm
[14:26:13] <ottomata>	 oh from an-coord
[14:26:15] <ottomata>	 right.
[14:26:31] <ottomata>	 joal try from ca-worker-1 then
[14:26:32] <joal>	 ottomata: Do we have another edge-node?
[14:26:34] <ottomata>	 might work from there
[14:26:36] <joal>	 Ahhh!
[14:28:46] <ottomata>	 looking into a refine user agent parse issue then will put attention back into presto :)
[14:28:58] <joal>	 np ottomata :)
[14:29:11] <joal>	 ottomata: While not knowing if ou were there, I loaded some data ;)
[14:47:29] <ottomata>	 oh boy
[14:54:42] <wikibugs>	 (03PS7) 10Mforns: Allow for custom transforms in DataFrameToDruid [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/477295 (https://phabricator.wikimedia.org/T210099)
[15:03:03] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10Product-Analytics: [BUG] User agent parsing error for MobileWikiAppSearch table - https://phabricator.wikimedia.org/T211833 (10Ottomata) This is indeed a bug!  Am investigating!  Not yet sure why this is happening.  Do we know when this started?
[15:14:09] <wikibugs>	 10Analytics-EventLogging, 10Analytics-Kanban: [EventLogging Sanitization] Enable older-than-90-day purging of unsanitized EL database (event) in Hive - https://phabricator.wikimedia.org/T209503 (10mforns) @nuria, yes the docs were updated a couple months ago to include that, see: https://wikitech.wikimedia.org...
[15:21:38] <wikibugs>	 10Analytics, 10Research, 10MW-1.33-notes (1.33.0-wmf.1; 2018-10-23): Many client side errors on citation data, significant percentages of data lost - https://phabricator.wikimedia.org/T206083 (10bmansurov)
[15:21:53] <wikibugs>	 10Analytics, 10Research, 10MW-1.33-notes (1.33.0-wmf.1; 2018-10-23): Many client side errors on citation data, significant percentages of data lost - https://phabricator.wikimedia.org/T206083 (10bmansurov) 05Open→03Resolved
[15:23:22] <wikibugs>	 10Analytics, 10Patch-For-Review: Add CitationUsage fields to EL purging white-list - https://phabricator.wikimedia.org/T205272 (10bmansurov) 05Open→03stalled Stalled on review.
[15:24:04] <wikibugs>	 (03PS8) 10Mforns: Allow for custom transforms in DataFrameToDruid [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/477295 (https://phabricator.wikimedia.org/T210099)
[15:24:40] <wikibugs>	 (03PS9) 10Mforns: Allow for custom transforms in DataFrameToDruid [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/477295 (https://phabricator.wikimedia.org/T210099)
[15:27:42] <milimetric>	 joal: job died again, but I noticed the two times I ran it as HDFS it says it was "Killed" and the time I ran it under my user it said "finished with failed status"
[15:27:50] <milimetric>	 any new thoughts on why?
[15:27:56] <milimetric>	 the shuffling thing you were saying yesterday?
[15:52:53] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10Product-Analytics, 10Reading-analysis: Final Vetting of Family Wide unique devices data - https://phabricator.wikimedia.org/T169550 (10Nuria)
[15:52:56] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Final steps to expose project family unique devices data - https://phabricator.wikimedia.org/T167539 (10Nuria) 05Open→03Resolved
[15:55:26] <nuria>	 ottomata: have a meeting but will help with UA issue after standup
[16:01:49] <wikibugs>	 10Analytics, 10Analytics-EventLogging: EventLogging client side serialization saves integer decimals as decimal-less numbers - https://phabricator.wikimedia.org/T211983 (10Ottomata)
[16:08:28] <wikibugs>	 (03PS5) 10Milimetric: Add tooltip Vue directive and use it [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/478824 (https://phabricator.wikimedia.org/T177950)
[16:10:30] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10Product-Analytics: [BUG] User agent parsing error for MobileWikiAppSearch table - https://phabricator.wikimedia.org/T211833 (10Ottomata) It isn't just useragent, recvfrom and seqid are also null.
[16:22:31] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10Product-Analytics: [BUG] User agent parsing error for MobileWikiAppSearch table - https://phabricator.wikimedia.org/T211833 (10Ottomata) FYI this started happening during the last refinery-source deployment around 2018-11-29T17:41:00Z.  I did make a change to the refinement...
[16:22:44] <wikibugs>	 (03CR) 10Milimetric: [C: 04-1] "I commented out the warning, otherwise we'd get many warnings until all tooltips are in place.  I suppose we can merge this and add more t" (033 comments) [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/478824 (https://phabricator.wikimedia.org/T177950) (owner: 10Milimetric)
[16:27:38] <wikibugs>	 (03CR) 10Nuria: "I think is fine to add more tooltips on a subsequent patch and merge this one, let's just not forget to do that." [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/478824 (https://phabricator.wikimedia.org/T177950) (owner: 10Milimetric)
[16:45:55] <wikibugs>	 (03CR) 10Milimetric: [C: 03+2] Add tooltip Vue directive and use it [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/478824 (https://phabricator.wikimedia.org/T177950) (owner: 10Milimetric)
[16:47:57] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats: Implement tooltips with async text load - https://phabricator.wikimedia.org/T211990 (10Milimetric) p:05Triage→03High
[16:49:47] <ottomata>	 joal:  yt?
[16:50:05] <ottomata>	 would love some dataframe cast help, am a little confused...
[16:56:48] <joal>	 ottomata: Heya ! Just here
[16:57:57] <joal>	 milimetric: a new instance I am testing- skewed-shuffling was definitely the problem, and actually not only with comment
[16:58:20] <joal>	 ottomata: joining the cave now if you wasnt
[17:01:09] <ottomata>	 ah sorry
[17:01:38] <nuria>	 ping fdans 
[17:01:45] <nuria>	 ping elukey 
[17:27:09] <milimetric>	 musikanimal: I'm sorry!  you should kill your top pages query, it has at least 2 major flaws
[17:27:28] <milimetric>	 the main one is limit 100 :)  which means it'll group and compute all the tops but only return the first 100
[17:27:37] <milimetric>	 I'll work on a better version and let you know
[17:28:20] <musikanimal>	 milimetric: oh I know :) it took 45 minutes to run. I've continued discussion at https://phabricator.wikimedia.org/T211827#4822756
[17:29:10] <musikanimal>	 I have a query there that in theory should work for us, but Hive doesn't like the syntax
[17:29:53] <musikanimal>	 there's probably an easier way
[17:48:47] <nuria>	 ottomata: let's batcave for bug on bc2 if you want?
[17:50:12] <ottomata>	 nuria:  am very close lemme keep hammering
[17:50:22] <nuria>	 ottomata: k
[17:51:36] <ottomata>	 it was caused by this change: https://github.com/wikimedia/analytics-refinery-source/commit/f26243d275d5b94c9151b22f0ca1ff06907e6dc7#diff-bad8f74fdd5bb84600d4fd9a89b789a3R657
[17:51:44] <ottomata>	 which happened a while ago, but wasn't yet deployed for refinery jobs
[17:51:49] <ottomata>	 (jar version hadn't been bumped)
[17:52:06] <ottomata>	 i think i know why, am adding a test and seeing what I can do about it...
[18:00:17] <nuria>	 ottomata: ok, will help test
[18:00:42] <nuria>	 musikanimal: can i help you with that?
[18:13:55] <wikibugs>	 10Analytics, 10Patch-For-Review: Add CitationUsage fields to EL purging white-list - https://phabricator.wikimedia.org/T205272 (10Miriam) @bmansurov there is no need for whitelisting the CitationUsage schema anymore, given recent discussions between Legal, Research and Analytics.
[18:14:25] <ottomata>	 nuria in batcave with joal now talking
[18:14:33] <ottomata>	 if you are interested
[18:14:42] <ottomata>	 i found out why it is happening, not sure yet what to do
[18:16:50] <nuria>	 ottomata: k
[18:17:49] <elukey>	 ottomata: https://www.confluent.io/confluent-community-license-faq (from Faidon)
[18:18:58] <elukey>	 also for the whole team --^
[18:23:05] <wikibugs>	 (03Abandoned) 10Bmansurov: Add CitationUsage and CitationUsagePageLoad to EL whitelist [analytics/refinery] - 10https://gerrit.wikimedia.org/r/462521 (https://phabricator.wikimedia.org/T205272) (owner: 10Bmansurov)
[18:23:28] <wikibugs>	 10Analytics, 10Patch-For-Review: Add CitationUsage fields to EL purging white-list - https://phabricator.wikimedia.org/T205272 (10bmansurov) 05stalled→03Invalid @Miriam OK, closing the task.
[18:26:50] * elukey off!
[18:28:31] <nuria>	 bmansurov: i am a little lost with the citation usage changes and retention, can we talk about thsoe when you have a minute?
[18:29:17] <wikibugs>	 10Analytics, 10Research, 10WMDE-Analytics-Engineering, 10User-Addshore, 10User-Elukey: Phase out and replace analytics-store (multisource) - https://phabricator.wikimedia.org/T172410 (10elukey) @Neil_P._Quinn_WMF Any thoughts about the above proposal? (I know that it will not solve all the problems, just...
[18:29:35] <DarTar>	 nuria: I think asking leila/miriam to document the proposal in writing, like you suggested, is a good idea
[18:30:04] <leila>	 nuria: the process is documented. miriam is updating one item in it and we will send it your way shortly.
[18:30:11] <leila>	 (saw your email asking for it.:)
[18:30:24] <nuria>	 leila: we are not whitelisting naa fields?
[18:30:26] <nuria>	 *any
[18:31:02] <leila>	 nuria: no. the plan is to drop those from eventLogging altogether. before doing so, we make a copy of all the data that we need, and that is the data that is cleared that we can keep for longer time for analysis.
[18:31:23] <leila>	 nuria: in that sense, there is no reason to create extra conditions on EventLogging end to whitelist items. 
[18:31:54] <nuria>	 leila: and why the  whitelist  does not work?
[18:32:03] <leila>	 nuria: I'm going to a meeting now. How about we send you the doc in less than an hour and then you and I can talk?
[18:32:37] <nuria>	 leila: sure
[18:41:12] <bmansurov>	 nuria: sure, I have time now
[18:58:19] <nuria>	 bmansurov: so you have decided to copy the  EL from EL tables data to a different location on stats machines, am i understanding this right?
[18:58:48] <bmansurov>	 nuria: not sure, I was out of loop. Miriam knows this.
[18:59:13] <bmansurov>	 nuria: I just abandoned the task because apparently it was not needed.
[19:18:30] <wikibugs>	 (03PS1) 10Ottomata: Normalize DataFrame before converting it to widen typed schema [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/479743 (https://phabricator.wikimedia.org/T211833)
[19:19:25] <wikibugs>	 (03PS2) 10Ottomata: Normalize DataFrame before converting it to widen typed schema [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/479743 (https://phabricator.wikimedia.org/T211833)
[19:20:15] <wikibugs>	 (03PS3) 10Ottomata: Normalize DataFrame before converting it to widen typed schema [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/479743 (https://phabricator.wikimedia.org/T211833)
[19:21:26] <wikibugs>	 (03PS4) 10Ottomata: Normalize DataFrame before converting it to widen typed schema [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/479743 (https://phabricator.wikimedia.org/T211833)
[19:23:02] <wikibugs>	 (03PS5) 10Ottomata: Normalize DataFrame before converting it to widen typed schema [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/479743 (https://phabricator.wikimedia.org/T211833)
[19:31:07] <ottomata>	 joal:  nuria quick +1 here so I can merge deploy and re refine that missing data?
[19:31:07] <ottomata>	 https://gerrit.wikimedia.org/r/#/c/analytics/refinery/source/+/479743/
[19:48:10] <nuria>	 ottomata: i really cannot CR the change without understanding better what is going on , i left CR up to joal, will look at test and code today
[19:48:33] <ottomata>	 its ok i have tested it
[19:48:44] <ottomata>	 joal did the code with me
[19:48:48] <ottomata>	 i'm going to self merge and deploy
[19:49:20] <wikibugs>	 (03CR) 10Ottomata: [C: 03+2] Normalize DataFrame before converting it to widen typed schema [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/479743 (https://phabricator.wikimedia.org/T211833) (owner: 10Ottomata)
[19:50:06] <joal>	 I was dining - I +2 indeed ottomata, nuria 
[19:50:15] <ottomata>	 thanks
[19:50:34] <ottomata>	 !log staring refinery release deploy process for refinery 0.0.82 to fix T211833
[19:50:36] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[19:50:37] <stashbot>	 T211833: [BUG] User agent parsing error for MobileWikiAppSearch table - https://phabricator.wikimedia.org/T211833
[20:07:55] <wikibugs>	 10Analytics, 10Operations, 10Research, 10Patch-For-Review, 10User-Banyek: Import recommendations into production database - https://phabricator.wikimedia.org/T208622 (10bmansurov) 05Open→03Resolved a:03bmansurov @Ottomata helped import the data we need for now. I'll follow up on open questions in o...
[20:22:06] <ottomata>	 mforns: , yt?
[20:22:10] <mforns>	 yep!
[20:22:15] <mforns>	 sup
[20:22:25] <ottomata>	 i need to rerefine pretty much everything since nov 29th
[20:22:35] <ottomata>	 which means we also should resanitize, ya?
[20:22:52] <mforns>	 ok, are you deploying refinery source?
[20:23:00] <ottomata>	 already done
[20:23:13] <mforns>	 ok
[20:23:24] <mforns>	 ottomata, yes, I will resanitize since then
[20:23:26] <ottomata>	 ok
[20:23:30] <ottomata>	 lemme do refine first
[20:23:32] <mforns>	 ottomata, all tables?
[20:24:03] <ottomata>	 ya, all recvFrom, seqId and userAgent fields are null since then
[20:24:08] <mforns>	 oh ok
[20:24:22] <ottomata>	 ok, so hey hgm
[20:24:29] <ottomata>	 does sanitize write a _SUCCESS flag?
[20:24:45] <ottomata>	 into the source dir?
[20:25:17] <ottomata>	 hm no it must be refine
[20:25:17] <mforns>	 ottomata, it uses refine, so I guess yes
[20:25:23] <ottomata>	 oh
[20:25:27] <mforns>	 let me check
[20:25:56] <ottomata>	 hm i don't know where it comes from..
[20:26:32] <ottomata>	 Refine uses _REFINED
[20:26:53] <mforns>	 oh
[20:27:33] <ottomata>	 ok so
[20:27:53] <ottomata>	 i'm going to remove all _REFINED flag files from data since 2018-11-29T17:00:00 onward
[20:28:03] <mforns>	 ok
[20:35:21] <ottomata>	 !log removing EventLogging Hive _REFINED flag files since 2018-11-29T17:00:00 to allow for re-refinement of data - 2018-11-29T17:00:00
[20:35:22] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[20:36:52] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10Product-Analytics, 10Patch-For-Review: [BUG] User agent parsing error for MobileWikiAppSearch table - https://phabricator.wikimedia.org/T211833 (10Tbayer) Thanks for the swift action on this! Let's make sure that other users of EL data become aware of the issue as well, e...
[20:44:35] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10Product-Analytics, 10Patch-For-Review: [BUG] User agent parsing error for MobileWikiAppSearch table - https://phabricator.wikimedia.org/T211833 (10Ottomata) Yup, backfilling now, will send email for sure.   This affects all top level table fields (ones in EventCapsule) th...
[20:45:10] <ottomata>	 !log re-refining all hive EventLogging tables since 2018-11-29T17:00:00. - T211833
[20:45:13] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[20:45:15] <stashbot>	 T211833: [BUG] User agent parsing error for MobileWikiAppSearch table - https://phabricator.wikimedia.org/T211833
[20:45:20] <ottomata>	 mforns:  this will probably take a while... :)
[20:45:26] <ottomata>	 w can probably re-sanitize on monday?
[20:48:35] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10DBA, 10Data-Services, and 3 others: Create materialized views on Wiki Replica hosts for better query performance - https://phabricator.wikimedia.org/T210693 (10Milimetric) @Banyek are you asking whether the views should have indices?  I'm not sure for the cloud users use...
[20:53:17] <nuria>	 ottomata: back
[20:53:39] <nuria>	 ottomata, mforns : does sanitize rely on _REFINED flags?
[20:53:50] <mforns>	 yes
[20:54:12] <mforns>	 why nuria?
[20:55:08] <nuria>	 mforns: cause if those are removed and we delete  all sanitized data from 2018-11-29
[20:55:17] <nuria>	 mforns: woudl the sanitization process not rerun?
[20:56:05] <mforns>	 nuria, only last couple days
[20:56:32] <mforns>	 the cron job that launches it has a since-until interval that spans 1 or 2 days IIRC
[20:56:33] <nuria>	 mforns: ah cause it is triggered via cron that  just looks back couple days?
[20:56:39] <mforns>	 yea
[20:57:05] <mforns>	 but I can delete since 2018-11-29 and backfill, no prob
[20:58:02] <nuria>	 ottomata: how did you re-run refined? 
[20:59:25] <ottomata>	 nuria:  https://phabricator.wikimedia.org/T211833#4824816
[21:00:00] <nuria>	 ottomata: i think i am getting confused with refined and sanitized, will removing REFINED flag will retrigger refine job?
[21:01:02] <nuria>	 mforns: i think more and more we are seeing that sanitization needs to be delayed from refinment
[21:01:15] <mforns>	 ottomata, re.sanitizing on monday, yes, will do that on monday
[21:01:31] <mforns>	 nuria, yes I agree
[21:01:39] <mforns>	 but I think we should have 2 sanitization
[21:01:50] <nuria>	 mforns: how so?
[21:02:01] <mforns>	 1 right away and 1 after say 80 days
[21:02:26] <nuria>	 mforns: why do we need the right away one?
[21:02:45] <ottomata>	 same answer for santizie
[21:02:48] <mforns>	 otherwise the event_sanitized database will not contain the last 3 months of data
[21:02:51] <ottomata>	 it only looks back some number of hours
[21:03:00] <nuria>	 mforns: i think that is fine
[21:03:07] <ottomata>	 so the --since flag in my command makes it look back father
[21:03:20] <nuria>	 mforns: i doubt anyone looks at it  for the last month
[21:03:25] <mforns>	 and if analysts want to query their tables historically until now, then they have to union 2 queries
[21:03:32] <nuria>	 ottomata: ahhh OK now i get it
[21:03:46] <mforns>	 nuria, both tilman and neil commented on the task asking explicitly for that
[21:04:06] <mforns>	 looking for it
[21:05:18] <mforns>	 nuria, https://phabricator.wikimedia.org/T209503#4751162
[21:06:45] <nuria>	 mforns: i see,  operationally we are seeing that probably would not work and having another process means another location to which data gets written
[21:07:23] <mforns>	 nuria, I think we can override event_sanitized no?
[21:07:58] <mforns>	 we can just execute the same job with -ignore-refine-flag no?
[21:08:47] <nuria>	 mforns: override? no
[21:09:05] <nuria>	 mforns: you cannot override in hive if you do not delete the data first
[21:09:36] <mforns>	 hmmm
[21:09:46] <nuria>	 mforns: there are no updates , you can add acolumns to a table and fill them with data going forward, but that's it
[21:10:12] <mforns>	 I knew about the updates, but thought we could replace the entire hourly partition
[21:10:20] <nuria>	 mforns: if you drop it yes
[21:10:47] <nuria>	 mforns: "insert overwrite"
[21:12:00] <mforns>	 nuria, ok, I was of the impression that refine job could replace an existing partition
[21:12:24] <mforns>	 then this would take some changes...
[21:12:36] <nuria>	 mforns:  i think refine it is just going to delete the data  and insert it again cc ottomata right?
[21:12:58] <mforns>	 nuria, well, then that would be perfect, that's OK
[21:13:08] <ottomata>	 its just going to overwrite the data
[21:14:43] <mforns>	 ok, then we just need a way to tell Refine to execute all targets in a given interval, regardless of _SANITIZED files
[21:14:53] <mforns>	 is that possible ottomata ?
[21:15:34] <mforns>	 just refine all targets, even if they are already sanitized?
[21:16:33] <mforns>	 nuria, in any case, we can push back the initial sanitization a couple days I guess
[21:16:36] <ottomata>	 mforns:  maybe we should add a --force flag
[21:16:53] <mforns>	 ottomata, aha
[21:17:07] <mforns>	 would be simple no?
[21:18:28] <ottomata>	 ya adding now
[21:18:32] <ottomata>	 ... :)
[21:19:34] <mforns>	 ottomata, no! I can do it when we decide to work on that!
[21:19:43] <mforns>	 I mean, if you want
[21:19:57] <ottomata>	 naw i've got code editor opened aand eveyrhing
[21:19:57] <ottomata>	 its ok
[21:19:58] <ottomata>	 actually
[21:20:00] <ottomata>	 to be consistent
[21:20:01] <mforns>	 ok
[21:20:09] <ottomata>	 do you think it would be better to have a --should_ignore_success flag?
[21:20:33] <mforns>	 to be like the --should_ignore_failure_flag?
[21:20:36] <ottomata>	 ya
[21:20:41] <mforns>	 maybe
[21:20:47] <mforns>	 dunno no preference
[21:20:47] <ottomata>	 it will be doing the same thing, except for considering the sucecss flag
[21:20:52] <mforns>	 haa
[21:20:56] <mforns>	 *aha
[21:21:16] <mforns>	 yes, sounds good!
[21:21:19] <nuria>	 ottomata, mforns : this also might be a stupid question
[21:21:46] <nuria>	 ottomata, mforns : but where in the refine code are we deleting  any data if it i s there? 
[21:22:10] <nuria>	 ottomata: cause a rerun with no refine needs to override the hourly partition, right? https://github.com/wikimedia/analytics-refinery-source/blob/master/refinery-job/src/main/scala/org/wikimedia/analytics/refinery/job/refine/Refine.scala
[21:23:11] <mforns>	 I don't know...
[21:23:15] <ottomata>	 nuria:  DataFrameToHive
[21:23:19] <ottomata>	 Line 173
[21:23:23] <ottomata>	 outputDf.df.write.mode("overwrite")
[21:23:32] <nuria>	 ottomata: ooohhhh
[21:23:53] <nuria>	 mforns: which means that sanitize will override if retrigger too no?
[21:24:07] <mforns>	 yes
[21:24:34] <mforns>	 I think that is good, because in case sanitization fails, we have 90 days to fix it
[21:24:57] <mforns>	 whereas, if we sanitize on the verge of raw data deletion, there's no backfilling possibilities
[21:25:18] <mforns>	 hmmm, actually...
[21:26:10] <nuria>	 mforns: yesss?
[21:26:13] <mforns>	 if we deploy a buggy version of sanitization, the second sanitization pass will corrupt the data without possibility of backfilling anyway
[21:26:23] <mforns>	 but....
[21:26:44] <nuria>	 mforns: sanitization is a "fixable" problem (when it comes to bus) on a 90 day window and that's it
[21:26:51] <nuria>	 *bugs
[21:27:03] <mforns>	 hmmmm
[21:27:32] <mforns>	 nuria, but if we re-sanitize very close to the 90 day limit, we might discover the bug too late
[21:27:34] <mforns>	 to backfill
[21:27:54] <mforns>	 maybe the second pass should not be very close to the 90 day limit
[21:28:02] <nuria>	 mforns: i think bugs on sanitize are not the target of thsi process, rather bugs on refine
[21:28:16] <mforns>	 right
[21:28:16] <nuria>	 mforns: so 1 pass right away and another one say 1 month later 
[21:28:23] <mforns>	 yes, this makes sense
[21:28:30] <nuria>	 mforns: ok, creating ticket
[21:29:11] <mforns>	 nuria, this also helps in late whitelist changes
[21:29:32] <mforns>	 so, every time someone creates a new EL schema, they can forget about the whitelist patch (even if documented)
[21:29:54] <mforns>	 this way they would have one month to work on the whitlist patch after first event comes in
[21:30:48] <mforns>	 or, a very common thing as well, they create the whitelist patch, but then the results are not what they expected, and have to modify it after couple weeks have passed
[21:31:30] <wikibugs>	 10Analytics, 10Analytics-Kanban: Sanitization should  be run a second time - https://phabricator.wikimedia.org/T212014 (10Nuria)
[21:31:50] <nuria>	 mforns: ya, totally: https://phabricator.wikimedia.org/T212014
[21:32:02] <nuria>	 mforns: given that overrides are handle is totally free
[21:32:09] <mforns>	 aha
[21:32:10] <nuria>	 mforns: task in kanban
[21:32:16] <mforns>	 thanks!
[21:32:22] <nuria>	 mforns: let's do a try out with this case 
[21:32:25] <mforns>	 well, there's the change ottomata is doing
[21:33:34] <nuria>	 mforns: let's not run re-sanitization by hand (cc ottomata ) but let's have a  "sanitization-swipper" cron do it  setting it up to look 1  month back, if we get it done next week it should have re-done everything on the 29th this month
[21:33:57] <mforns>	 makes sense
[21:34:03] <nuria>	 mforns: last thing to do on thsi was turning these crons into systemd timers right?
[21:34:04] <mforns>	 I can work on that next week
[21:34:12] <mforns>	 yes
[21:34:27] <nuria>	 mforns: ok, we also need to deploy the eventlogging sanitization monitor too
[21:34:39] <mforns>	 well, there's a couple changes to be code-reviewed and maybe there's things to adjust?
[21:34:51] <mforns>	 also need to deploy and check there are no problems in prod
[21:34:58] <mforns>	 yes
[21:36:07] <nuria>	 cc ottomata see above, we think of setting up a sanitization-swipper cron that will take care of re-sanitization 1 month after the fact, so rather than re-sanitize on Monday let's let this swipper cron do it 
[21:36:19] <nuria>	 ottomata: so sanitization will happen twice
[21:36:30] <nuria>	 ottomata: once right away and once some weeks after 
[21:36:45] <nuria>	 ottomata: the second time will override the 1st one
[21:37:25] <nuria>	 musikanimal: your FAVORITE page on top 3 https://en.wikipedia.org/wiki/User:West.andrew.g/2018_Popular_pages cc milimetric 
[21:37:55] <musikanimal>	 haha
[21:38:19] * musikanimal Main Page and `-` don't count, so my fav is actually #1 !!!
[21:38:29] <nuria>	 musikanimal: we are going to get rid of that one with the bot classifier, in fact let me check my test data and see where it lands
[21:38:37] <nuria>	 musikanimal: JAJAJa
[21:39:26] <nuria>	 musikanimal: SO TRUE
[21:39:45] <musikanimal>	 :)
[21:40:08] <ottomata>	 oh kkk
[21:40:21] <musikanimal>	 Also, I have for a long time been suspicious of [[Darth Vader]]
[21:41:41] <joal>	 ok - the actor trick alone seems not enough for mediawiki-history
[21:41:55] <joal>	 Will add the comment one on monday and test
[21:41:55] <nuria>	 musikanimal: mmmmmm
[21:42:03] <musikanimal>	 every month people report it as a false positive in Topviews, but I was unable to confirm this from my limited research
[21:42:09] <joal>	 Gone for tonight, have a good weekend team
[21:42:19] <nuria>	 musikanimal: how do people report it?
[21:42:31] <nuria>	 joal: ciao joal , thank for your explanations!
[21:42:51] <joal>	 nuria: Hopefully it'll work at some point :)
[21:43:21] <musikanimal>	 there's a little report system thingy on https://tools.wmflabs.org/topviews/ You click the X to hide a page, then use the "report false positive" link towards the top, above the list of "Excluded pages"
[21:44:15] <nuria>	 musikanimal: nice, real nice
[21:44:36] <musikanimal>	 my normal tactic is to check location of traffic. If it's like 90% in the same city (often not a sizable city), I go ahead and mark it as confirmed which hides it from Topviews. This was not the case with Darth Vader, last I checked
[21:46:40] <nuria>	 musikanimal: ya, on the classifier I am doing similar thing: if i entity is requesting pages  too fast  (i think i put 15 times in a minute)  it is just fake bot traffic 
[21:46:51] <wikibugs>	 10Analytics, 10Research, 10WMDE-Analytics-Engineering, 10User-Addshore, 10User-Elukey: Phase out and replace analytics-store (multisource) - https://phabricator.wikimedia.org/T172410 (10Neil_P._Quinn_WMF) @elukey, thank you for that detailed explanation! As always, thank you for stewarding our infrastruc...
[21:48:11] <nuria>	 musikanimal: my ideas are more about entities than pages but i will ping you  once i check these two pages, let me see what my classification says for these pages
[21:48:34] <musikanimal>	 cool, thank you!
[21:52:22] <wikibugs>	 (03PS1) 10Ottomata: Add --ignore_done_flag option to Refine to ease re-running jobs [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/479851
[21:53:21] <ottomata>	 mforns_afk:  nuria^
[21:54:11] <wikibugs>	 (03PS2) 10Ottomata: Add --ignore_done_flag option to Refine to ease re-running jobs [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/479851
[21:54:36] <wikibugs>	 (03CR) 10Ottomata: "tested!" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/479851 (owner: 10Ottomata)
[22:07:11] <wikibugs>	 (03CR) 10Nuria: [C: 03+2] Add --ignore_done_flag option to Refine to ease re-running jobs [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/479851 (owner: 10Ottomata)
[22:08:14] <nuria>	 milimetric: back to dashiki extension changes,
[22:12:46] <wikibugs>	 (03Merged) 10jenkins-bot: Add --ignore_done_flag option to Refine to ease re-running jobs [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/479851 (owner: 10Ottomata)
[22:12:50] <wikibugs>	 10Analytics, 10Wikipedia-iOS-App-Backlog, 10iOS-app-Bugs, 10iOS-app-feature-Analytics, 10iOS-app-v6.2-Beluga-On-A-Pogo-Stick: Many errors on    "MobileWikiAppiOSSearch"  and  "MobileWikiAppiOSUserHistory" - https://phabricator.wikimedia.org/T207424 (10chelsyx)
[22:16:34] <wikibugs>	 10Analytics, 10Developer-Advocacy, 10Product-Analytics, 10Documentation: Develop EventLogging schema for documentation feedback gadget - https://phabricator.wikimedia.org/T211638 (10srishakatux) 05Open→03Resolved Thanks @fdans!   Schema is up here: https://meta.wikimedia.org/wiki/Schema:UserFeedback.
[22:34:31] <wikibugs>	 10Analytics, 10Developer-Advocacy, 10Product-Analytics, 10Documentation: Develop EventLogging schema for documentation feedback gadget - https://phabricator.wikimedia.org/T211638 (10Nuria) @srishakatux something to think about is overall participation versus overall impressions of survey. Example:  "how ma...
[23:21:12] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10DBA, 10Data-Services, and 3 others: Create materialized views on Wiki Replica hosts for better query performance - https://phabricator.wikimedia.org/T210693 (10bd808) >>! In T210693#4824836, @Milimetric wrote: > I'm not sure for the cloud users use case, you'd have to loo...