[03:19:26] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [07:19:26] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [08:31:27] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Add cl_timestamp_id index to categorylinks table - https://phabricator.wikimedia.org/T399249#11022811 (10Marostegui) [09:39:02] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Add cl_timestamp_id index to categorylinks table - https://phabricator.wikimedia.org/T399249#11022999 (10Marostegui) [09:45:19] 14Analytics, 06Data-Engineering, 06Data-Engineering-Icebox: Count the number of video plays - https://phabricator.wikimedia.org/T198628#11023012 (10AndrewTavis_WMDE) Thanks for all the information, @TheDJ :) Bringing @Ben.buchenau in as well as we've been discussing this. Does seem like we're reliant on `pre... [09:47:09] 10Data-Engineering (Q4 2025 April 1st - June 30th): Airflow skips canary-event tasks - https://phabricator.wikimedia.org/T380836#11023019 (10Antoine_Quhen) 05Open→03Resolved Running the following script, I have checked we didn't meet this problem for the last 4 months (the maximum time Airflow keeps its... [10:27:50] 06Data-Engineering, 06Growth-Team, 10GrowthExperiments-NewcomerTasks, 10Discovery-Search (2025.07.04 - 2025.07.25), and 2 others: Error for mediawiki.cirrussearch-request: '' should NOT have additional properties - https://phabricator.wikimedia.org/T399965#11023206 (10Michael) We seem to be only very incid... [11:19:26] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [12:18:06] 06Data-Engineering, 10Event-Platform: [SPIKE] Use Flink for batch backfilling - https://phabricator.wikimedia.org/T324108#11023754 (10Ottomata) 05Open→03Declined no work on this, and should be possible if we ever need to do this. [12:21:09] 06Data-Engineering, 10Event-Platform: mediawiki-event-enrichment: changes to test image seem to be ignored in CI - https://phabricator.wikimedia.org/T340195#11023790 (10Ottomata) @gmodena is this still an issue? [12:22:45] 06Data-Engineering, 10Event-Platform: Add $comment and $performer to ArticleRevisionVisibilitySet params - https://phabricator.wikimedia.org/T321411#11023791 (10Ottomata) 05Open→03Declined Won't do. Domain Events FTW. [12:24:41] 06Data-Engineering, 07Epic, 10Event-Platform: [Event Platform] Flink Operations - https://phabricator.wikimedia.org/T328561#11023806 (10Ottomata) 05Open→03Resolved a:03Ottomata closing epic. One subtask remains but it can be treated alone. [12:25:13] 06Data-Engineering, 06Growth-Team, 10GrowthExperiments-NewcomerTasks, 10Discovery-Search (2025.07.04 - 2025.07.25), and 2 others: Error for mediawiki.cirrussearch-request: '' should NOT have additional properties - https://phabricator.wikimedia.org/T399965#11023811 (10Michael) >>! In T399965#11023206, @Mic... [12:25:44] 06Data-Engineering, 10Event-Platform: Support NULL values in RowData in eventutilities - https://phabricator.wikimedia.org/T328211#11023812 (10Ottomata) 05Open→03Declined Unlikely to do any Flink SQL work. If this changes we can reopen. [12:26:10] 06Data-Engineering, 10Event-Platform: Document and Promote Image Suggestions Feedback > Cassandra Flink Job - https://phabricator.wikimedia.org/T316112#11023833 (10Ottomata) 05Open→03Declined one day maybe... [12:29:28] 06Data-Engineering, 10Image-Suggestions, 10Event-Platform: Refactor Image Suggestions Feedback > Cassandra Flink Job and Deploy to DSE k8s - https://phabricator.wikimedia.org/T329524#11023849 (10Ottomata) 05Open→03Declined one day... [12:33:35] 06Data-Engineering, 10Event-Platform: [TEMPLATE] Onboard request for APPLICATION NAME to Event Platform - https://phabricator.wikimedia.org/T346207#11023880 (10Ottomata) 05Open→03Declined unused? [12:34:49] 14Analytics-Radar, 06Data-Engineering, 06Data-Engineering-Radar, 10MediaWiki-Recent-changes, and 3 others: Remove deprecated RCFeedEngine support - https://phabricator.wikimedia.org/T250628#11023886 (10Ottomata) 05Open→03Resolved a:03Ottomata this looks done? reopen if incorrect. [12:35:14] 06Data-Engineering: [session length] Investigate slight drop at sessions of 30 minutes or more - https://phabricator.wikimedia.org/T280254#11023890 (10Ottomata) [12:36:05] 06Data-Engineering, 10Event-Platform: mediawiki-event-enrichment deployment process should include producing an event in staging and verifying success - https://phabricator.wikimedia.org/T341138#11023897 (10Ottomata) →14Duplicate dup:03T347472 [12:36:07] 06Data-Engineering, 10Event-Platform: [NEEDS GROOMING] stream processing: we should have automated integration tests on staging - https://phabricator.wikimedia.org/T347472#11023899 (10Ottomata) [12:38:44] 06Data-Engineering, 06Discovery-Search, 06serviceops-radar, 10Event-Platform: [Event Platform] [NEEDS GROOMING] Store Flink HA metadata in Zookeeper - https://phabricator.wikimedia.org/T331283#11023924 (10Ottomata) p:05High→03Medium [12:39:26] 06Data-Engineering, 06tech-decision-forum, 10Event-Platform: MediaWiki Event Carried State Transfer - Problem Statement - https://phabricator.wikimedia.org/T291120#11023925 (10Ottomata) p:05High→03Medium [12:40:21] 14Analytics, 06Data-Engineering, 10Event-Platform: EventStreams sending same data over and over (page links change) - https://phabricator.wikimedia.org/T290211#11023926 (10Ottomata) p:05High→03Low [12:40:43] 06Data-Engineering, 10Event-Platform: Refine drops $schema field values - https://phabricator.wikimedia.org/T255818#11023927 (10Ottomata) 05Open→03Resolved a:03Ottomata [12:43:19] 06Data-Engineering, 10Event-Platform: Refine drops $schema field values - https://phabricator.wikimedia.org/T255818#11023936 (10Ottomata) It's fixed! ` presto:event> select _schema from mediawiki_page_change_v1 where month=7 and year=2025 and day=21 limit 10; _schema ---------------------------... [13:34:39] 14Analytics, 06Data-Engineering, 10Event-Platform, 07Wikimedia-Performance-recommendation: Avoid extra HTTPS connections for most Event Platform beacons - https://phabricator.wikimedia.org/T263049#11024146 (10Ottomata) @phuedx @dr0ptp4kt I assume the new beacon endpoint (`/beacon/v2/events` ?) exists now?... [14:19:01] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10DPE-Mediawiki-Content: Remove code and tables related to deprecated mediawiki_wikitext_history and mediawiki_wikitext_current - https://phabricator.wikimedia.org/T396031#11024246 (10xcollazo) [14:27:17] 10Analytics-Canonical-Data, 10Movement-Insights (FY25-26 H1): Create a canonical dataset for incubating wikis - https://phabricator.wikimedia.org/T393075#11024304 (10CMyrick-WMF) p:05Medium→03Low a:03CMyrick-WMF [14:27:58] 10Analytics-Canonical-Data, 10Movement-Insights (FY25-26 H1): Provide ISO 639 language codes in canonical wiki dataset - https://phabricator.wikimedia.org/T346855#11024309 (10CMyrick-WMF) p:05Medium→03Low a:03CMyrick-WMF [14:55:13] 10Analytics-Canonical-Data, 06Research, 10Movement-Insights (FY25-26 H1): Create a canonical dataset for incubating wikis - https://phabricator.wikimedia.org/T393075#11024457 (10Miriam) [14:55:52] 10Analytics-Canonical-Data, 06Research, 10Movement-Insights (FY25-26 H1): Provide ISO 639 language codes in canonical wiki dataset - https://phabricator.wikimedia.org/T346855#11024459 (10Miriam) [15:17:43] 06Data-Engineering, 06Data-Engineering-Radar, 10AbuseFilter, 07Schema-change, and 2 others: AbuseFilter abuse_filter_log table: Store IP addresses as hex values - https://phabricator.wikimedia.org/T395612#11024585 (10OKryva-WMF) p:05Triage→03Medium [15:19:26] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [16:37:17] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10Event-Platform, 13Patch-For-Review: Update event-producing tools to overwrite `meta.dt` - https://phabricator.wikimedia.org/T376026#11025019 (10Ottomata) p:05Triage→03Medium [17:38:23] 10Data-Engineering (Q4 2025 April 1st - June 30th), 13Patch-For-Review: Facilitate automatic artifact cache warming for airflow-dags artifacts - https://phabricator.wikimedia.org/T392244#11025288 (10amastilovic) 05Open→03Resolved [17:53:41] !log manual run pageview_actor.hql for pageview_actor_hourly__compute_pageview_actor_hourly__20250720 for hour 3 [17:53:44] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [18:47:39] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10Event-Platform, 13Patch-For-Review: Add alerting to eventbus and eventgate for drastic changes in event rate production. - https://phabricator.wikimedia.org/T398437#11025616 (10Ottomata) @gmodena +1 on both patches. I did not deeply review the alert queri... [19:05:18] 06Data-Engineering: ERROR AsyncEventQueue: Listener DatahubSparkListener threw an exception - https://phabricator.wikimedia.org/T400207#11025703 (10dr0ptp4kt) [19:08:01] 06Data-Engineering: ERROR AsyncEventQueue: Listener DatahubSparkListener threw an exception - https://phabricator.wikimedia.org/T400207#11025711 (10dr0ptp4kt) [19:12:12] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10Event-Platform, 13Patch-For-Review: Update event-producing tools to overwrite `meta.dt` - https://phabricator.wikimedia.org/T376026#11025718 (10Ottomata) Deployed eventgate in beta, looks good. [19:19:27] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [19:41:14] !log deploying eventgate-logging-external and eventgate-analytics-external to pick up meta.dt change - T376026 [19:41:18] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [19:41:19] T376026: Update event-producing tools to overwrite `meta.dt` - https://phabricator.wikimedia.org/T376026 [20:02:35] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10Event-Platform, 13Patch-For-Review: Update event-producing tools to overwrite `meta.dt` - https://phabricator.wikimedia.org/T376026#11025860 (10Ottomata) I deployed eventgate-logging-external and eventgate-analytics-external in codfw. Along the way, I not... [20:07:19] 10Data-Engineering (Q4 2025 April 1st - June 30th), 06Experimentation Lab, 10Event-Platform: EventGate: Add Prometheus metric for hoisting errors - https://phabricator.wikimedia.org/T398922#11025886 (10Ottomata) a:05tchin→03Ottomata I'm in this code for another task so I can do this. [20:17:16] 10Data-Engineering (Q4 2025 April 1st - June 30th), 06Experimentation Lab, 10Event-Platform: EventGate: Add Prometheus metric for hoisting errors - https://phabricator.wikimedia.org/T398922#11025892 (10Ottomata) Because HoistingError extends from ValidationError, I think they are currently being counted as V... [20:17:47] 10Data-Engineering (Q4 2025 April 1st - June 30th), 06Experimentation Lab, 10Event-Platform: EventGate: Add Prometheus metric for hoisting errors - https://phabricator.wikimedia.org/T398922#11025906 (10Ottomata) p:05Triage→03High [22:43:38] 10Data-Engineering (Q4 2025 April 1st - June 30th): test_produced_by_config SLA miss configured to be too small for upstream dataset run time - https://phabricator.wikimedia.org/T388861#11026210 (10amastilovic) I did some preliminary investigation of this issue, and it seems that the SLA on the `test_produced_by... [23:19:26] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem