[06:45:20] good morning :) [06:45:28] I am re-running the failed refine hours [06:50:27] elukey: o/ [06:55:17] hola hola [07:09:07] RECOVERY - Check the last execution of monitor_refine_eventlogging_legacy_failure_flags on an-launcher1002 is OK: OK: Status of the systemd unit monitor_refine_eventlogging_legacy_failure_flags https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [09:58:09] (03PS4) 10Fdans: Add filter/split component to Wikistats [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/613114 (https://phabricator.wikimedia.org/T249758) [09:59:59] (03CR) 10jerkins-bot: [V: 04-1] Add filter/split component to Wikistats [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/613114 (https://phabricator.wikimedia.org/T249758) (owner: 10Fdans) [10:00:14] omg what now [10:00:37] jenkins doesn't like you [10:04:26] elukey: not even jenkins likes me [10:12:35] oh wait I'm such a dingdong [10:12:45] I was only running 2 tests :( [10:12:49] great monday vibes [10:13:27] ahahahah [10:25:14] (03PS5) 10Fdans: Add filter/split component to Wikistats [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/613114 (https://phabricator.wikimedia.org/T249758) [10:27:25] (03CR) 10jerkins-bot: [V: 04-1] Add filter/split component to Wikistats [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/613114 (https://phabricator.wikimedia.org/T249758) (owner: 10Fdans) [10:31:12] (03CR) 10Fdans: "recheck" [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/613114 (https://phabricator.wikimedia.org/T249758) (owner: 10Fdans) [10:37:16] * elukey lunch! [12:43:31] helloo team :] [12:46:12] o/ [13:24:29] 10Analytics, 10Product-Analytics: NULL-values for useragent column in event.searchsatisfaction - https://phabricator.wikimedia.org/T259944 (10Ottomata) Backfill successful. ` 20/08/09 21:35:39 INFO Refine: Successfully refined 103 of 103 dataset partitions into table `event`.`SearchSatisfaction` (total # refi... [13:45:57] 10Analytics, 10Product-Analytics: NULL-values for useragent column in event.searchsatisfaction - https://phabricator.wikimedia.org/T259944 (10Ottomata) Ah of course. This is not related to the array of structs bug described in T259924. This is instead because the Refine transform function that adds the legac... [13:51:31] 10Analytics, 10Patch-For-Review, 10User-Elukey: Test if Hue can run with Python3 - https://phabricator.wikimedia.org/T233073 (10elukey) After some battling with Debian packages, I created https://github.com/cloudera/hue/issues/1239 since it is not clear to me what is the best build procedure. [13:51:43] 10Analytics-Clusters, 10Patch-For-Review, 10User-Elukey: Test if Hue can run with Python3 - https://phabricator.wikimedia.org/T233073 (10elukey) [13:59:39] (03PS1) 10Ottomata: refine - Add legacy useragent column if field exists in event schema or in Hive [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/619298 (https://phabricator.wikimedia.org/T259944) [14:10:03] (03PS2) 10Ottomata: refine - Add legacy useragent column if field exists in event schema or in Hive [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/619298 (https://phabricator.wikimedia.org/T259944) [14:28:49] 10Analytics-Radar, 10Datasets-General-or-Unknown, 10Product-Analytics, 10Structured-Data-Backlog: Set up generation of JSON dumps for Wikimedia Commons - https://phabricator.wikimedia.org/T259067 (10Cparle) Is that adequate for you @ArielGlenn ? [14:52:58] (03CR) 10Ottomata: Use properties to configure compiler source and target versions. (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/615485 (https://phabricator.wikimedia.org/T258699) (owner: 10Gehel) [14:54:16] (03PS3) 10Ottomata: refine - Add legacy useragent column if field exists in event schema or in Hive [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/619298 (https://phabricator.wikimedia.org/T259944) [14:56:17] (03CR) 10Ottomata: [C: 03+1] Introduce Takari Maven Wrapper. [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/615481 (https://phabricator.wikimedia.org/T258699) (owner: 10Gehel) [15:08:20] mforns: this is the one ready for review: v [15:08:21] https://gerrit.wikimedia.org/r/c/analytics/refinery/source/+/619298 [15:08:36] thx ottomata, will look after meetings [15:13:40] a-team: sorry yall I’m off this week and forgot it’s my ops week [15:13:57] I can switch with anyone, sorry for the late notice [15:31:50] 10Analytics, 10Operations, 10SRE-Access-Requests, 10Patch-For-Review: Requesting access to production shell for Denny Vrandecic - https://phabricator.wikimedia.org/T259388 (10mforns) ping @akosiaris? :] [15:32:10] 10Analytics-Radar, 10Operations, 10SRE-Access-Requests, 10Patch-For-Review: Requesting access to production shell for Denny Vrandecic - https://phabricator.wikimedia.org/T259388 (10mforns) [15:32:42] 10Analytics, 10Event-Platform, 10Product-Infrastructure-Data: Streams with empty configs should be rendered as {} in the JSON returned by StreamConfig API - https://phabricator.wikimedia.org/T259917 (10mforns) a:03fdans [15:32:57] 10Analytics, 10Analytics-Kanban, 10Event-Platform, 10Product-Infrastructure-Data: Streams with empty configs should be rendered as {} in the JSON returned by StreamConfig API - https://phabricator.wikimedia.org/T259917 (10mforns) [15:33:18] 10Analytics, 10Analytics-Wikistats: Wikistats New Feature - https://phabricator.wikimedia.org/T258996 (10mforns) 05Open→03Invalid [15:34:27] 10Analytics-Radar, 10Datasets-General-or-Unknown, 10Product-Analytics, 10Structured-Data-Backlog: Set up generation of JSON dumps for Wikimedia Commons - https://phabricator.wikimedia.org/T259067 (10ArielGlenn) >>! In T259067#6373045, @Cparle wrote: > Is that adequate for you @ArielGlenn ? That should be... [15:35:40] 10Analytics, 10Analytics-Kanban, 10Event-Platform, 10Patch-For-Review: Instrumentation development environment on EventGate server - https://phabricator.wikimedia.org/T259202 (10mforns) [15:35:51] 10Analytics, 10Analytics-Kanban, 10Event-Platform, 10Patch-For-Review: Instrumentation development environment on EventGate server - https://phabricator.wikimedia.org/T259202 (10mforns) p:05Triage→03High [15:35:53] 10Analytics, 10Analytics-EventLogging, 10Event-Platform: Prevent schema creation in meta for eventlogging schemas - https://phabricator.wikimedia.org/T259201 (10Ottomata) [15:35:55] 10Analytics, 10Better Use Of Data, 10Event-Platform, 10Product-Infrastructure-Data, 10Goal: BUOD: Require that all new schema/instruments are created with the MEP system - https://phabricator.wikimedia.org/T259157 (10Ottomata) [15:36:17] 10Analytics, 10Analytics-EventLogging, 10Event-Platform: Prevent schema creation in meta for eventlogging schemas - https://phabricator.wikimedia.org/T259201 (10Ottomata) [15:37:31] 10Analytics, 10VPS-Projects, 10Puppet: Puppet failing on wikistats.analytics.eqiad.wmflabs due to statistics::user - https://phabricator.wikimedia.org/T259307 (10mforns) p:05Triage→03Medium Maybe a good task for onboarding. [15:38:45] 10Analytics, 10Analytics-Kanban, 10Design-Research: Setup and integrate analytics for Design Research Website - https://phabricator.wikimedia.org/T259322 (10mforns) a:03Nuria [15:41:12] 10Analytics: Check home/HDFS leftovers of demon - https://phabricator.wikimedia.org/T259585 (10mforns) From backlog grooming: We think this can be deleted now, after 1 year. [15:41:31] 10Analytics: Check home/HDFS leftovers of demon - https://phabricator.wikimedia.org/T259585 (10mforns) a:03fdans [15:42:06] 10Analytics-Radar, 10MediaWiki-REST-API, 10Platform Team Sprints Board (Sprint 1), 10Platform Team Workboards (Green): Unify access log schema for Action API and API Gateway/REST API - https://phabricator.wikimedia.org/T259736 (10mforns) [15:43:07] 10Analytics, 10Voice & Tone: Rename geoeditors_blacklist_country - https://phabricator.wikimedia.org/T259804 (10mforns) p:05Triage→03Low From backlog grooming [15:48:06] 10Analytics, 10Analytics-Data-Quality: page_id is null where it shouldn't be in mediawiki history - https://phabricator.wikimedia.org/T259823 (10mforns) p:05Triage→03Medium Maybe a task for Lex. [15:48:45] 10Analytics, 10Analytics-Kanban, 10Event-Platform, 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services): Set up Jenkins maven release job for wikimedia-event-utilities like analytics/refinery/source - https://phabricator.wikimedia.org/T259898 (10mforns) p:05Triage→03High [15:49:38] 10Analytics, 10Patch-For-Review: HiveExtensions.convertToSchema does not properly convert arrays of structs - https://phabricator.wikimedia.org/T259924 (10mforns) p:05Triage→03High a:03Ottomata [15:49:47] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: HiveExtensions.convertToSchema does not properly convert arrays of structs - https://phabricator.wikimedia.org/T259924 (10mforns) [15:50:14] 10Analytics, 10Product-Analytics, 10Patch-For-Review: NULL-values for useragent column in event.searchsatisfaction - https://phabricator.wikimedia.org/T259944 (10mforns) p:05Triage→03High a:03Ottomata [15:50:25] 10Analytics, 10Analytics-Kanban, 10Product-Analytics, 10Patch-For-Review: NULL-values for useragent column in event.searchsatisfaction - https://phabricator.wikimedia.org/T259944 (10mforns) [15:51:06] 10Analytics, 10Analytics-EventLogging, 10Event-Platform: Prevent schema creation in meta for eventlogging schemas - https://phabricator.wikimedia.org/T259201 (10mforns) p:05Triage→03Medium [15:55:09] 10Analytics: Remove request for font.googleapis.com from analytics.wikimedia.org - https://phabricator.wikimedia.org/T182804 (10mforns) [15:55:11] 10Analytics, 10Analytics-Kanban: analytics.wikimedia.org TLC - https://phabricator.wikimedia.org/T253393 (10mforns) [15:56:07] 10Analytics: Whitelist analytics.wikimedia.org and stats.wikimedia.org in ad blockers - https://phabricator.wikimedia.org/T182816 (10mforns) 05Open→03Declined From grooming: we haven't been able to do this. [15:57:04] 10Analytics: [Wikistats Design] integrate browser dashboard data into wikistats - https://phabricator.wikimedia.org/T198333 (10mforns) p:05Triage→03Low [15:59:44] 10Analytics, 10Analytics-Dashiki: Provide filterable line graph for browser-family/browser-major - https://phabricator.wikimedia.org/T150713 (10mforns) 05Open→03Declined For internal data we have Turnilo now, and we don't have plans to work on our external data browser dashboards. [16:00:03] 10Analytics: (Marker) Everything Above is for Deprioritized column - https://phabricator.wikimedia.org/T247253 (10mforns) 05Open→03Invalid [16:00:25] 10Analytics: Open source Spark DataFrame to hive refine job - https://phabricator.wikimedia.org/T191034 (10mforns) 05Open→03Declined [16:01:06] 10Analytics: Easter Egg: wikistats classic style on wikistats 2.0 - https://phabricator.wikimedia.org/T177408 (10mforns) p:05Medium→03Low [16:02:22] 10Analytics: Host API for token persistence dataset - https://phabricator.wikimedia.org/T164280 (10mforns) 05Open→03Declined From team grooming. Declining, as we haven't had any buy in for this task. Please, reopen if necessary. [16:03:00] 10Analytics, 10Patch-For-Review: Sort inconsistency in AQS timestamp behavior - https://phabricator.wikimedia.org/T160311 (10mforns) p:05Medium→03Low [16:03:35] 10Analytics, 10Data-release: Wikipedia Clickstream dataset. Programmatic Access - https://phabricator.wikimedia.org/T134231 (10mforns) Maybe a good one for Lex. [16:04:26] 10Analytics: Prototype counting of requests with real time (streaming data) - https://phabricator.wikimedia.org/T159264 (10mforns) From grooming: Closing this, as we have many other open prototypes. [16:04:42] 10Analytics: Prototype counting of requests with real time (streaming data) - https://phabricator.wikimedia.org/T159264 (10mforns) 05Open→03Declined [16:04:45] 10Analytics-EventLogging, 10Analytics-Kanban, 10Event-Platform, 10Goal, and 3 others: Modern Event Platform (TEC2) - https://phabricator.wikimedia.org/T185233 (10mforns) [16:08:22] 10Analytics: Better publishing of Annotations about Data Issues - https://phabricator.wikimedia.org/T142408 (10mforns) p:05Medium→03Lowest [16:09:33] 10Analytics: Add jobs for druid compaction for pageviews data set - https://phabricator.wikimedia.org/T164500 (10mforns) 05Open→03Resolved a:03mforns Already done. [16:11:00] 10Analytics: Describe threat model for sanitized pageview data {mole} - https://phabricator.wikimedia.org/T131158 (10mforns) 05Open→03Declined We have rejected anonymization as a viable strategy. https://github.com/wikimedia/analytics-refinery/tree/master/oozie/pageview/druid/monthly [16:11:10] 10Analytics, 10Browser-Support-Opera: Split opera mini in proxy or turbo mode - https://phabricator.wikimedia.org/T138505 (10mforns) 05Open→03Invalid No longer relevant. [16:15:02] 10Analytics, 10Analytics-Wikistats: Add edit/upload distinction to mediawiki history pipeline - https://phabricator.wikimedia.org/T178017 (10mforns) 05Open→03Declined We acknowledge we need upload data, but that should come from the instrumentation. [16:19:02] 10Analytics, 10Scoring-platform-team, 10articlequality-modeling, 10Spike, 10artificial-intelligence: [Spike] Store article quality data inside hadoop and make AQS outputs a public API - https://phabricator.wikimedia.org/T164377 (10mforns) 05Open→03Declined From team grooming: Closing this task becaus... [16:20:19] 10Analytics: Create new table for 'referer' aggregated data - https://phabricator.wikimedia.org/T112284 (10mforns) p:05Low→03Medium [16:23:43] 10Analytics, 10MediaWiki-Releasing: Create dashboard showing MediaWiki tarball download statistics - https://phabricator.wikimedia.org/T119772 (10mforns) 05Open→03Declined Declining, because we have the datasets from the pingback extension. Please, reopen if necessary. [16:25:25] 10Analytics: Transform and Import Qualtrics Survey data - https://phabricator.wikimedia.org/T184626 (10mforns) 05Open→03Declined We won't be able to work on this any soon. [16:25:58] 10Analytics: Create ops dashboard with info like ipv6 traffic split - https://phabricator.wikimedia.org/T138396 (10mforns) 05Open→03Declined Also, not working on that anytime soon. [16:26:59] 10Analytics-Radar, 10Operations, 10SRE-Access-Requests, 10Patch-For-Review: Requesting access to production shell for Denny Vrandecic - https://phabricator.wikimedia.org/T259388 (10Vgutierrez) @akosiaris is on vacations, I'll handle this ASAP [16:27:26] 10Analytics, 10good first task: Reportupdater: do not write execution control files in source directories - https://phabricator.wikimedia.org/T173604 (10mforns) p:05Low→03Lowest [16:27:36] 10Analytics-Radar, 10Operations, 10SRE-Access-Requests, 10Patch-For-Review: Requesting access to production shell for Denny Vrandecic - https://phabricator.wikimedia.org/T259388 (10Vgutierrez) a:05DVrandecic→03Vgutierrez [16:30:53] 10Analytics, 10Analytics-Wikistats: Render Wikistats graphs server-side, so that they cam be embedded on-wikis - https://phabricator.wikimedia.org/T178016 (10mforns) [16:31:51] 10Analytics, 10Analytics-Dashiki: Paginate table-timeseries visualization - https://phabricator.wikimedia.org/T191270 (10mforns) 05Open→03Declined We're most probably not going to work on this, please reopen if needed. [16:32:34] 10Analytics: Investigate adding user-friendly testing functionality to Reportupdater - https://phabricator.wikimedia.org/T156523 (10mforns) p:05Low→03Lowest [16:33:46] 10Analytics, 10Performance-Team (Radar): Eventlogging client needs to support batching of events for offline use case (also better Perf overall) - https://phabricator.wikimedia.org/T162308 (10mforns) 05Open→03Declined Declined, we already have batching of events. Though, not for offline purposes. Please,... [16:35:05] 10Analytics, 10Analytics-Wikistats: Add overall ORES scores to Wikistats - https://phabricator.wikimedia.org/T178019 (10mforns) 05Open→03Declined Declining, because ORES scores are not representative of whole projects. Please, reopen if necessary. [16:36:25] 10Analytics-Radar, 10Operations, 10SRE-Access-Requests, 10Patch-For-Review: Requesting access to production shell for Denny Vrandecic - https://phabricator.wikimedia.org/T259388 (10Nuria) @DVrandecic will also need a kerberos password [16:36:33] 10Analytics, 10Analytics-Wikistats: Deploy Wikistats and analytics.wikimedia.org via SCAP - https://phabricator.wikimedia.org/T170429 (10mforns) 05Open→03Declined Probably not accurate any more, closing. Please reopen if appropriate. [16:38:21] 10Analytics: Productionize analysis of editcount vs per_user_revision_count - https://phabricator.wikimedia.org/T168648 (10mforns) @JAllemandou Does this still apply? [16:38:48] 10Analytics: Combine Hive Year / Month / Day / Hour partitions into ISO date string - https://phabricator.wikimedia.org/T177097 (10mforns) 05Open→03Declined We decided not to do this. Closing [16:41:16] 10Analytics: Remove "bot" from metrics/pageviews/per-article - https://phabricator.wikimedia.org/T178448 (10mforns) 05Open→03Resolved a:03mforns Already done, closing: https://wikimedia.org/api/rest_v1/#/Pageviews%20data/get_metrics_pageviews_per_article__project___access___agent___article___granularity___... [16:42:44] 10Analytics: Put data needed for edits metrics through Event Bus into HDFS - https://phabricator.wikimedia.org/T131782 (10mforns) 05Open→03Resolved a:03mforns Already done. [16:42:46] 10Analytics-Kanban: Implement Pages Created & Count of Edits full vertical slice - https://phabricator.wikimedia.org/T131779 (10mforns) [16:43:50] 10Analytics, 10Platform Team Workboards (Initiatives): reportupdater Pingback reports are broken and need to be refactored - https://phabricator.wikimedia.org/T246154 (10CCicalese_WMF) [16:44:11] 10Analytics, 10MediaWiki-Releasing: Create dashboard showing MediaWiki tarball download statistics - https://phabricator.wikimedia.org/T119772 (10CCicalese_WMF) Note that the pingback reports are still broken: https://phabricator.wikimedia.org/T246154. [16:54:26] 10Analytics, 10Analytics-Kanban, 10Product-Analytics, 10Patch-For-Review: NULL-values for useragent column in event.searchsatisfaction - https://phabricator.wikimedia.org/T259944 (10nettrom_WMF) Thanks for taking care of this so quickly, @Ottomata, very much appreciated! > BTW, is user_agent_map non Null?... [16:55:38] 10Analytics-Radar, 10Operations, 10SRE-Access-Requests, 10Patch-For-Review: Requesting access to production shell for Denny Vrandecic - https://phabricator.wikimedia.org/T259388 (10Vgutierrez) 05Open→03Resolved >>! In T259388#6373672, @Nuria wrote: > @DVrandecic will also need a kerberos password ` v... [17:53:01] * elukey afk! [17:56:27] (03CR) 10Mforns: "Left some comments, let me know if they make sense." (034 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/619298 (https://phabricator.wikimedia.org/T259944) (owner: 10Ottomata) [18:04:29] (03CR) 10Ottomata: refine - Add legacy useragent column if field exists in event schema or in Hive (034 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/619298 (https://phabricator.wikimedia.org/T259944) (owner: 10Ottomata) [18:04:47] (03PS4) 10Ottomata: refine - Add legacy useragent column if field exists in event schema or in Hive [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/619298 (https://phabricator.wikimedia.org/T259944) [18:13:07] (03CR) 10Ottomata: [C: 03+2] refine - Add legacy useragent column if field exists in event schema or in Hive [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/619298 (https://phabricator.wikimedia.org/T259944) (owner: 10Ottomata) [18:14:03] 10Analytics, 10Growth-Team, 10Product-Analytics (Kanban): Growth: validate that data is purged after 270 days - https://phabricator.wikimedia.org/T249666 (10nettrom_WMF) [18:14:26] 10Analytics, 10Growth-Team (Current Sprint), 10Product-Analytics (Kanban): Growth: validate that data is purged after 270 days - https://phabricator.wikimedia.org/T249666 (10nettrom_WMF) [18:22:23] 10Analytics, 10Growth-Team (Current Sprint), 10Product-Analytics (Kanban): Growth: validate that data is purged after 270 days - https://phabricator.wikimedia.org/T249666 (10nettrom_WMF) I ran `SHOW PARTITIONS event_sanitized.homepagemodule` on 2020-08-07 and again today (2020-08-10). After the first run, I... [19:05:55] will be back al ittle later [19:30:08] (03CR) 10Gehel: Use properties to configure compiler source and target versions. (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/615485 (https://phabricator.wikimedia.org/T258699) (owner: 10Gehel) [19:48:32] b [21:33:13] 10Analytics, 10MediaWiki-REST-API, 10Patch-For-Review, 10Platform Team Sprints Board (Sprint 1), and 2 others: System administrator reviews API usage by client - https://phabricator.wikimedia.org/T251812 (10Pchelolo) Ok... unexpected complication - envoy JSON access_log formatter currently only supports si... [21:57:40] 10Analytics, 10MediaWiki-REST-API, 10Patch-For-Review, 10Platform Team Sprints Board (Sprint 1), and 2 others: System administrator reviews API usage by client - https://phabricator.wikimedia.org/T251812 (10Pchelolo) Ok. I've submitted https://github.com/envoyproxy/envoy/issues/12582 to support nested form... [21:58:59] 10Analytics, 10Analytics-Kanban, 10Design-Research: Setup and integrate analytics for Design Research Website - https://phabricator.wikimedia.org/T259322 (10Nuria) Can you please write on ticket what the top domain would be? [22:49:09] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Update PageviewDefinition to only include /api/rest_v1/page/mobile-html requests with X-Analytics: pageview=1 in pageviews - https://phabricator.wikimedia.org/T257860 (10Nuria) Without correction: ` select count(*), access_method, year, month, day, hour... [22:52:51] 10Analytics, 10Analytics-Kanban, 10Platform Team Workboards (Initiatives): Design Document that proposes an alternative architecture for historic data endpoints - https://phabricator.wikimedia.org/T241184 (10Nuria) 05Open→03Resolved [22:52:53] 10Analytics: MW REST API Historical Data Endpoint Needs - https://phabricator.wikimedia.org/T240387 (10Nuria) [23:10:49] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: HiveExtensions.convertToSchema does not properly convert arrays of structs - https://phabricator.wikimedia.org/T259924 (10Ottomata) AHHHH RATS. I don't think Refine can do this, at least not with reading the incoming data with the merged Hive schema. `AR...