[01:53:44] (03CR) 10Milimetric: [C: 03+2] Base class checkArgsSize [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/681624 (owner: 10Awight) [02:02:15] (03Merged) 10jenkins-bot: Base class checkArgsSize [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/681624 (owner: 10Awight) [02:02:42] (03PS3) 10Milimetric: Use base class methods to check argument type and convert [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/681389 (owner: 10Awight) [02:04:34] (03PS2) 10Milimetric: [WIP] POC: loading cassandra directly from spark [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/686629 [02:06:40] (03CR) 10Milimetric: [C: 03+2] Base class checkArgPrimitive [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/681625 (owner: 10Awight) [02:14:25] (03Merged) 10jenkins-bot: Base class checkArgPrimitive [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/681625 (owner: 10Awight) [02:15:16] (03CR) 10Milimetric: [C: 03+2] Use base class methods to check argument type and convert [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/681389 (owner: 10Awight) [02:22:37] (03Merged) 10jenkins-bot: Use base class methods to check argument type and convert [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/681389 (owner: 10Awight) [02:36:25] 10Analytics-Clusters, 10Analytics-Kanban, 10Patch-For-Review: Review and improve Oozie authorization permissions - https://phabricator.wikimedia.org/T262660 (10razzi) 05Open→03Resolved [04:26:26] PROBLEM - Check unit status of monitor_refine_event_sanitized_analytics_delayed on an-launcher1002 is CRITICAL: CRITICAL: Status of the systemd unit monitor_refine_event_sanitized_analytics_delayed https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [07:20:05] 10Analytics, 10DBA, 10Event-Platform, 10WMF-Architecture-Team, 10Services (later): Consistent MediaWiki state change events | MediaWiki events as source of truth - https://phabricator.wikimedia.org/T120242 (10Joe) >>! In T120242#6945584, @Ottomata wrote: > Another idea that may not be feasible: Would it... [09:02:19] 10Analytics, 10LDAP-Access-Requests, 10SRE, 10CommRel-Specialists-Support (Apr-Jun-2021): Please grant CRS access to Superset/Turnilo - trizek - https://phabricator.wikimedia.org/T282772 (10Elitre) [09:03:03] 10Analytics, 10LDAP-Access-Requests, 10SRE, 10CommRel-Specialists-Support (Apr-Jun-2021): Please grant CRS access to Superset/Turnilo - Johan - https://phabricator.wikimedia.org/T282773 (10Elitre) [09:03:51] 10Analytics, 10LDAP-Access-Requests, 10SRE, 10CommRel-Specialists-Support (Apr-Jun-2021): Please grant CRS access to Superset/Turnilo - Keegan - https://phabricator.wikimedia.org/T282774 (10Elitre) [09:05:19] 10Analytics, 10LDAP-Access-Requests, 10SRE, 10CommRel-Specialists-Support (Apr-Jun-2021): Please grant CRS access to Superset/Turnilo (deadline EOD Monday 17) - https://phabricator.wikimedia.org/T282589 (10Elitre) [09:11:58] 10Analytics, 10LDAP-Access-Requests, 10SRE, 10CommRel-Specialists-Support (Apr-Jun-2021): Please grant CRS access to Superset/Turnilo (deadline EOD Monday 17) - https://phabricator.wikimedia.org/T282589 (10Elitre) [09:12:19] 10Analytics, 10LDAP-Access-Requests, 10SRE, 10CommRel-Specialists-Support (Apr-Jun-2021): Please grant CRS access to Superset/Turnilo - Elitre - https://phabricator.wikimedia.org/T282776 (10Elitre) [09:12:49] 10Analytics, 10LDAP-Access-Requests, 10SRE, 10CommRel-Specialists-Support (Apr-Jun-2021): Please grant CRS access to Superset/Turnilo (deadline EOD Monday 17) - https://phabricator.wikimedia.org/T282589 (10elukey) Added `elitre` to the `wmf` LDAP group. [09:14:22] 10Analytics, 10LDAP-Access-Requests, 10SRE, 10CommRel-Specialists-Support (Apr-Jun-2021): Please grant CRS access to Superset/Turnilo (deadline EOD Monday 17) - https://phabricator.wikimedia.org/T282589 (10elukey) [09:14:55] 10Analytics, 10LDAP-Access-Requests, 10SRE, 10CommRel-Specialists-Support (Apr-Jun-2021): Please grant CRS access to Superset/Turnilo - Elitre - https://phabricator.wikimedia.org/T282776 (10elukey) 05Open→03Resolved a:03elukey Followed up on slack, added `elitre` to `wmf`. [09:14:58] 10Analytics, 10LDAP-Access-Requests, 10SRE, 10CommRel-Specialists-Support (Apr-Jun-2021): Please grant CRS access to Superset/Turnilo (deadline EOD Monday 17) - https://phabricator.wikimedia.org/T282589 (10elukey) [09:16:50] 10Analytics, 10LDAP-Access-Requests, 10SRE, 10CommRel-Specialists-Support (Apr-Jun-2021): Please grant CRS access to Superset/Turnilo - Johan - https://phabricator.wikimedia.org/T282773 (10elukey) 05Open→03Resolved a:03elukey Followed up with @Elitre on slack, also verified that user `johan` was assi... [09:16:54] 10Analytics, 10LDAP-Access-Requests, 10SRE, 10CommRel-Specialists-Support (Apr-Jun-2021): Please grant CRS access to Superset/Turnilo (deadline EOD Monday 17) - https://phabricator.wikimedia.org/T282589 (10elukey) [09:21:21] 10Analytics, 10LDAP-Access-Requests, 10SRE, 10CommRel-Specialists-Support (Apr-Jun-2021): Please grant CRS access to Superset/Turnilo - Keegan - https://phabricator.wikimedia.org/T282774 (10elukey) 05Open→03Resolved a:03elukey @wikimedia.org email for LDAP uid `keegan`, plus I have followed up with @... [09:21:24] 10Analytics, 10LDAP-Access-Requests, 10SRE, 10CommRel-Specialists-Support (Apr-Jun-2021): Please grant CRS access to Superset/Turnilo (deadline EOD Monday 17) - https://phabricator.wikimedia.org/T282589 (10elukey) [09:27:03] 10Analytics, 10LDAP-Access-Requests, 10SRE, 10CommRel-Specialists-Support (Apr-Jun-2021): Please grant CRS access to Superset/Turnilo - trizek - https://phabricator.wikimedia.org/T282772 (10elukey) 05Open→03Resolved a:03elukey Added `trizek` to `wmf`. The email associated with the account was not @wi... [09:27:06] 10Analytics, 10LDAP-Access-Requests, 10SRE, 10CommRel-Specialists-Support (Apr-Jun-2021): Please grant CRS access to Superset/Turnilo (deadline EOD Monday 17) - https://phabricator.wikimedia.org/T282589 (10elukey) [09:33:58] 10Analytics, 10LDAP-Access-Requests, 10SRE, 10CommRel-Specialists-Support (Apr-Jun-2021): Please grant CRS access to Superset/Turnilo (deadline EOD Monday 17) - https://phabricator.wikimedia.org/T282589 (10elukey) Note: the superset dashboards may not be all accessible if all users are also part of the `an... [10:00:29] elukey, joal: given that we're gonna have to do a full reimport anyway, and also given that the repair will take probably weeks, would it be a worthwhile experiment to truncate the existing tables on the new cluster, repair with a lot less data and then see if the jobs succeed? [10:02:33] hnowlan: +1 seems a good idea [10:38:13] (03CR) 10Gergő Tisza: [C: 03+2] Create structured_task/article/link_suggestion_interaction schema [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/681052 (https://phabricator.wikimedia.org/T278177) (owner: 10Kosta Harlan) [10:38:51] (03Merged) 10jenkins-bot: Create structured_task/article/link_suggestion_interaction schema [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/681052 (https://phabricator.wikimedia.org/T278177) (owner: 10Kosta Harlan) [11:41:04] !log running truncate "local_group_default_T_pageviews_per_article_flat".data; on aqs1012 [11:41:06] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [12:32:02] the new Cassandra cluster is doing a whole load of anticompactions now and it'll probably take a good long while [12:36:43] ack [12:57:09] 10Analytics, 10DBA, 10Event-Platform, 10WMF-Architecture-Team, 10Services (later): Consistent MediaWiki state change events | MediaWiki events as source of truth - https://phabricator.wikimedia.org/T120242 (10Ottomata) > it would make the database availability depend on the availability of eventgate, and... [13:09:02] ottomata: the deletion script is still dropping partitions O.o! [13:09:24] the mediawiki_* ones are almost done, though... [13:11:13] mforns: wow yeah taking a while [13:11:25] also, there's an alert about monitor_refine_event_sanitized_analytics_delayed [13:11:28] i'm looking intio it [13:11:45] its a casing issue, it looks like 3-4 hours of those two tables somehow got missed in my sanitize backfill when we changed casing [13:11:55] oh [13:11:57] the data exists but it is in the upper case event tables [13:12:34] gonna try to refine sanitize those...but i think i have to remember/find the old job config that used dir regex insetead of hive metastore to find the data [13:12:40] not sure [13:13:54] ottomata: those schemas are the ones I backfilled for FR, maybe while you were changing CamelCase to lowercase? [13:14:01] oh yu backfilled them? [13:14:17] yes, a while ago [13:14:22] huh, maybe. [13:14:24] like one month ago? [13:14:33] or more [13:14:48] mobilewikiappiosfeed ? [13:14:52] wait [13:14:53] and wikipediaportal [13:14:55] yes [13:15:02] mobilewikiappiosfeed for FR? [13:15:23] yes I think so [13:16:07] ottomata: https://phabricator.wikimedia.org/T273246 [13:16:57] But the only thing I did was backfill sanitization, so if something it would have added some directories in CamelCase to event_sanitized [13:17:08] oh! [13:17:19] but not readded CamelCase directories to event [13:17:55] huh mayyyybe somehow these got lost when we did the table renames??? [13:31:29] interesting mforns i think i just was able to rerun the refine_sanitize for that time period and it is doing them [13:31:32] without any special flags [13:31:42] ok [13:31:53] do we have to change the names to lowercase? [13:33:05] no...i think that maybe if we had waited the delayed job would have just done it on its owwn [13:33:15] but the monitor checks a wider period than the refine job does [13:33:18] so it found them first [13:47:42] (03CR) 10Milimetric: "Reporting back about the performance test with pageviews_per_article for one day. I kind of don't understand the logs... So I ran it wit" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/686629 (owner: 10Milimetric) [14:33:36] hey folks :) [14:34:16] for some reason I got a reminder that I was in ops week but I see Marcel listed [14:34:19] weird [15:11:28] elukey: hm... did you get the reminder today? [15:16:18] mforns: I did! weird, but it maybe a one off with my gcal, if it re-happens I'll dive deep [15:17:05] elukey: it might be because today razzi's week starts, and it was previously yours, maybe it's a legacy gcal thing [15:18:21] ack ack [15:32:59] mforns: buod backlog grooming? [15:36:17] 10Analytics, 10Event-Platform, 10Product-Analytics: Augment Hive event data with normalized host info from meta.domain - https://phabricator.wikimedia.org/T251320 (10ldelench_wmf) [15:44:53] 10Analytics, 10LDAP-Access-Requests, 10SRE, 10CommRel-Specialists-Support (Apr-Jun-2021): Please grant CRS access to Superset/Turnilo (deadline EOD Monday 17) - https://phabricator.wikimedia.org/T282589 (10mpopov) Thanks so much @elukey you're the best! [15:49:10] 10Analytics, 10Better Use Of Data, 10Product-Data-Infrastructure, 10Metrics-Platform (Metrics-Platform-MVP-Release-1): Define acceptable usage of the `meta` object in event schemas - https://phabricator.wikimedia.org/T273293 (10DAbad) Based on recent schema discussions I think we can close this ticket. @jl... [15:49:20] 10Analytics, 10Better Use Of Data, 10Product-Data-Infrastructure, 10Metrics-Platform (Metrics-Platform-MVP-Release-1): Define acceptable usage of the `meta` object in event schemas - https://phabricator.wikimedia.org/T273293 (10DAbad) 05Open→03Resolved [15:51:43] 10Analytics, 10Better Use Of Data, 10Event-Platform, 10Metrics-Platform: wgEventStreams (EventStreamConfig) should support per wiki overrides - https://phabricator.wikimedia.org/T277193 (10ldelench_wmf) [15:53:25] 10Analytics, 10Better Use Of Data, 10Event-Platform, 10Metrics-Platform: wgEventStreams (EventStreamConfig) should support per wiki overrides - https://phabricator.wikimedia.org/T277193 (10DAbad) If we don't do this then we can't change the setting for different wikis. [15:55:44] 10Analytics, 10Better Use Of Data, 10Event-Platform, 10Metrics-Platform: wgEventStreams (EventStreamConfig) should support per wiki overrides - https://phabricator.wikimedia.org/T277193 (10DAbad) [16:06:31] 10Analytics-Clusters, 10Analytics-Kanban: Re-add disk to an-worker1100 - https://phabricator.wikimedia.org/T281427 (10razzi) 05Open→03Resolved I checked and the disk is filling up; this can be closed. [16:36:48] 10Analytics-Radar, 10GrowthExperiments, 10Growth-Team (Current Sprint), 10MW-1.36-notes (1.36.0-wmf.30; 2021-02-09): eventgate_validation_error for NewcomerTask, HomepageTask, and HomepageVisit schemas - https://phabricator.wikimedia.org/T273700 (10Milimetric) [16:39:05] 10Analytics, 10Analytics-Kanban: [Newpyter] Can't install 'haven' package with conda R but can with system R - https://phabricator.wikimedia.org/T282262 (10Milimetric) p:05Triage→03High a:03Ottomata [16:40:21] 10Analytics, 10Analytics-Kanban: [Newpyter] Conda stacked environment overwrites TAR environment variable - https://phabricator.wikimedia.org/T282491 (10Milimetric) p:05Triage→03High a:03Ottomata [16:42:21] 10Analytics-EventLogging, 10Analytics-Radar, 10Metrics-Platform, 10Product-Data-Infrastructure, 10Vector (Vector (Tracking)): EventLogging revision popup gets hidden behind content in Vector - https://phabricator.wikimedia.org/T282550 (10Milimetric) @jlinehan and/or @Mholloway want to take a look? If no... [16:43:05] 10Analytics, 10Analytics-Kanban, 10Event-Platform: WMDEBanner* Event Platform Migration - https://phabricator.wikimedia.org/T282562 (10Milimetric) p:05Triage→03High [16:47:20] 10Analytics, 10Analytics-EventLogging, 10Wikimedia-Developer-Portal, 10Documentation: Clean up EventLogging Schema: pages on meta - https://phabricator.wikimedia.org/T282584 (10Milimetric) p:05Triage→03High [16:48:59] 10Analytics-EventLogging, 10Analytics-Radar, 10Metrics-Platform, 10Product-Data-Infrastructure, 10Vector (Vector (Tracking)): EventLogging revision popup gets hidden behind content in Vector - https://phabricator.wikimedia.org/T282550 (10Jdlrobson) The issue here is that the .mw-json-schema-code-samples... [16:49:30] 10Analytics: Superset query timeouts for charts using Druid table - https://phabricator.wikimedia.org/T282618 (10Milimetric) cc-ing @JAllemandou, who said wanted to look at it, we'll triage with him Monday when he's back from vacation [16:52:21] 10Analytics, 10Analytics-Kanban: Superset Presto LIMIT >10000 error - https://phabricator.wikimedia.org/T282632 (10Milimetric) p:05Triage→03High a:03Milimetric [16:54:57] 10Analytics, 10Research: Adding data from centralauth to the lake and the mediawiki_history dataset - https://phabricator.wikimedia.org/T282657 (10Milimetric) p:05Triage→03Medium This would have to happen after data governance, so any help before that is appreciated (I can review patches to the pipeline an... [16:57:13] 10Analytics, 10LDAP-Access-Requests, 10SRE, 10CommRel-Specialists-Support (Apr-Jun-2021): Please grant CRS access to Superset/Turnilo (deadline EOD Monday 17) - https://phabricator.wikimedia.org/T282589 (10Milimetric) p:05Triage→03High a:03elukey [16:58:49] 10Analytics, 10Analytics-Kanban: Missing data in virtualpageview_hourly table since April 15, 2021 - https://phabricator.wikimedia.org/T282710 (10Milimetric) p:05Triage→03Unbreak! a:03mforns Culprit is uppercase mismatch, so druid jobs weren't finding the data: https://github.com/wikimedia/analytics-refi... [17:50:44] (03PS1) 10Mforns: Fix broken jobs due to lowercasing of event db data directories [analytics/refinery] - 10https://gerrit.wikimedia.org/r/690607 (https://phabricator.wikimedia.org/T282710) [19:01:03] (03CR) 10Ottomata: [C: 03+1] Fix broken jobs due to lowercasing of event db data directories [analytics/refinery] - 10https://gerrit.wikimedia.org/r/690607 (https://phabricator.wikimedia.org/T282710) (owner: 10Mforns) [19:26:42] PROBLEM - Check unit status of produce_canary_events on an-launcher1002 is CRITICAL: CRITICAL: Status of the systemd unit produce_canary_events https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [19:34:25] (03CR) 10Mforns: [V: 03+2 C: 03+2] "Given we wrote this in pairs, and we got an extra +1, the jobs are already running successfully, I'm merging this!" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/690607 (https://phabricator.wikimedia.org/T282710) (owner: 10Mforns) [19:38:24] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Missing data in virtualpageview_hourly table since April 15, 2021 - https://phabricator.wikimedia.org/T282710 (10mforns) This has been fixed and the jobs are back-filling both Hive and Druid (Turnilo). It will take a couple days to catch up, though. Thanks... [19:45:44] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Missing data in virtualpageview_hourly table since April 15, 2021 - https://phabricator.wikimedia.org/T282710 (10cchen) Thanks, @mforns @Milimetric ! [20:10:59] 10Analytics, 10Better Use Of Data, 10Event-Platform, 10Product-Analytics, and 2 others: EventLogging MEP Upgrade Phase 3 (Stream cc-ing) - https://phabricator.wikimedia.org/T256165 (10mpopov) 05Open→03Declined We abandoned the idea of stream cc-ing; this idea is vaguely present in some of the conceptua... [20:11:23] 10Analytics, 10Better Use Of Data, 10Performance-Team, 10Product-Analytics, and 2 others: Switch mw.user.sessionId back to session-cookie persistence - https://phabricator.wikimedia.org/T223931 (10mpopov) a:05mpopov→03None [20:17:19] 10Analytics-Radar, 10Better Use Of Data, 10Event-Platform, 10Metrics-Platform, and 2 others: Explore sending batches of events from EPC libraries - https://phabricator.wikimedia.org/T239996 (10ldelench_wmf) [20:17:29] 10Analytics-Radar, 10Better Use Of Data, 10Event-Platform, 10Metrics-Platform, and 2 others: Explore sending batches of events from EPC libraries - https://phabricator.wikimedia.org/T239996 (10ldelench_wmf) a:05mpopov→03jlinehan [20:20:49] 10Analytics-Radar, 10Better Use Of Data, 10Event-Platform, 10Metrics-Platform, 10Product-Data-Infrastructure: Explore sending batches of events from EPC libraries - https://phabricator.wikimedia.org/T239996 (10mpopov) [20:21:55] (03PS1) 10GoranSMilovanovic: T259105 [analytics/wmde/WD/WikidataAnalytics] - 10https://gerrit.wikimedia.org/r/690692 [20:22:08] (03CR) 10GoranSMilovanovic: [V: 03+2 C: 03+2] T259105 [analytics/wmde/WD/WikidataAnalytics] - 10https://gerrit.wikimedia.org/r/690692 (owner: 10GoranSMilovanovic) [20:37:31] 10Analytics, 10WMDE-Analytics-Engineering, 10WMDE-New-Editors-Banner-Campaigns: Drop old WMDEBanner events from Hive - https://phabricator.wikimedia.org/T281300 (10mforns) The data older than 90 days has been deleted. Cheers! [20:38:09] 10Analytics, 10Analytics-Kanban, 10WMDE-Analytics-Engineering, 10WMDE-New-Editors-Banner-Campaigns: Drop old WMDEBanner events from Hive - https://phabricator.wikimedia.org/T281300 (10mforns) [20:39:10] 10Analytics, 10Analytics-Kanban: Switch off skipTrash for some data purging - https://phabricator.wikimedia.org/T270431 (10mforns) This is deployed, and done. [20:40:37] 10Analytics, 10Better Use Of Data, 10Event-Platform, 10Product-Infrastructure-Team-Backlog, 10Epic: Event Platform Client Libraries - https://phabricator.wikimedia.org/T228175 (10Mholloway)