[02:36:58] 10Data-Engineering (Q1 FY25/26 July 1st - September 30th): Fix CommonsCategoryGraphBuilder to reflect latest changes to categorylinks table - https://phabricator.wikimedia.org/T404735#11192350 (10amastilovic) The `2025-08` backfill run of the DAG has completed successfully, and judging by data sizes on HDFS I'd... [06:39:27] 10Data-Engineering (Q1 FY25/26 July 1st - September 30th), 06Discovery-Search, 10Wikidata, 10Wikidata-Query-Service, and 2 others: [EPIC] Upgrade flink jobs to java 17 - https://phabricator.wikimedia.org/T404340#11192610 (10dcausse) [06:49:02] 10Data-Engineering (Q1 FY25/26 July 1st - September 30th), 06Discovery-Search, 10Wikidata, 10Wikidata-Query-Service, and 2 others: [EPIC] Upgrade flink jobs to java 17 - https://phabricator.wikimedia.org/T404340#11192631 (10dcausse) >>! In T404340#11191436, @gmodena wrote: > We need to spike readyness of `... [06:56:48] 06Data-Engineering, 06Data-Engineering-Radar, 06Discovery-Search, 06Infrastructure-Foundations, and 3 others: Elasticsearch dependency upgrade in spicerack - https://phabricator.wikimedia.org/T390860#11192639 (10RKemper) Have pushed out various improvements to the code. Still much to do on the unit test si... [07:22:20] 10Data-Engineering-Roadmap, 06Discovery-Search, 10DPE-Mediawiki-Content, 07Epic, 13Patch-For-Review: EPIC: Update flink jobs to support Flink 1.20 - https://phabricator.wikimedia.org/T376812#11192692 (10dcausse) [07:25:53] 10Data-Engineering (Q1 FY25/26 July 1st - September 30th), 06Discovery-Search, 10Wikidata, 10Wikidata-Query-Service, and 2 others: [EPIC] Upgrade flink jobs to java 17 - https://phabricator.wikimedia.org/T404340#11192696 (10dcausse) [07:44:06] 10Data-Engineering (Q1 FY25/26 July 1st - September 30th), 06Discovery-Search, 10Wikidata, 10Wikidata-Query-Service, and 2 others: [EPIC] Upgrade flink jobs to java 17 - https://phabricator.wikimedia.org/T404340#11192725 (10gmodena) >>! In T404340#11192631, @dcausse wrote: >>>! In T404340#11191436, @gmoden... [07:58:24] 06Data-Engineering, 06Abstract Wikipedia team, 06Movement-Insights, 10Wikifunctions: Bug: Cassandra Unique Devices data quality issue: mobile data - https://phabricator.wikimedia.org/T370551#11192756 (10JAllemandou) I completely have this in mind. I think this bug is a duplicate of T299559. I had planned t... [08:02:37] 10Data-Engineering (Q1 FY25/26 July 1st - September 30th): Update unique_devices tables to add the `access_method` field - https://phabricator.wikimedia.org/T401666#11192761 (10JAllemandou) [08:02:50] 06Data-Engineering, 06Data-Engineering-Icebox, 14Data-Engineering-Kanban, 10Data-Engineering-Wikistats, 06Product-Analytics: Wikistats reports no mobile unique devices for Wikidata, MediaWiki.org, Wikifunctions - https://phabricator.wikimedia.org/T299559#11192764 (10JAllemandou) [08:03:45] 06Data-Engineering, 06Data-Engineering-Icebox, 14Data-Engineering-Kanban, 10Data-Engineering-Wikistats, 06Product-Analytics: Wikistats reports no mobile unique devices for Wikidata, MediaWiki.org, Wikifunctions - https://phabricator.wikimedia.org/T299559#11192768 (10JAllemandou) [08:03:46] 06Data-Engineering, 06Abstract Wikipedia team, 06Movement-Insights, 10Wikifunctions: Bug: Cassandra Unique Devices data quality issue: mobile data - https://phabricator.wikimedia.org/T370551#11192766 (10JAllemandou) →14Duplicate dup:03T299559 [10:26:12] !log Deploying Refinery at 5feb53f9 for https://gerrit.wikimedia.org/r/c/analytics/refinery/+/1188748 and https://gerrit.wikimedia.org/r/c/analytics/refinery/+/1188291 [10:26:14] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [10:34:51] !log temporarily disabled puppet on A:hadoop in order to test https://gerrit.wikimedia.org/r/c/operations/puppet/+/1189221 for T404871 [10:34:53] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [10:34:54] T404871: Add resource preemption to Hadoop Yarn scheduler - https://phabricator.wikimedia.org/T404871 [10:50:16] !log Deployed refinery to HDFS [10:50:17] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [11:39:42] 10Data-Engineering (Q1 FY25/26 July 1st - September 30th), 06Data-Persistence, 10Data-Persistence-Design-Review, 06Growth-Team, and 3 others: Data Persistence Design Review: Improve Tone Suggested Edits newcomer task - https://phabricator.wikimedia.org/T401021#11193231 (10Michael) >>! In T401021#11190742,... [12:06:17] FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=codfw%2Bprometheus/k8s&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [12:21:17] RESOLVED: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=codfw%2Bprometheus/k8s&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [13:01:18] 10Data-Engineering (Q1 FY25/26 July 1st - September 30th): Optimize metrics computation for the MW Content Pipeline - https://phabricator.wikimedia.org/T401010#11193476 (10Antoine_Quhen) `sql SELECT count(1) FROM wmf_data_ops.data_quality_alerts.snapshots; ` 101 318 snapshots `python jvm = sc._jvm path_str = "/... [13:17:19] 06Data-Engineering, 06Data-Engineering-Radar, 10CirrusSearch, 10Structured Data Engineering, and 4 others: Migrate image recommendation to use page_weighted_tags_changed stream - https://phabricator.wikimedia.org/T372912#11193550 (10dcausse) >>! In T372912#11181194, @pfischer wrote: > Would moving the DAG... [13:22:51] 10Data-Engineering (Q1 FY25/26 July 1st - September 30th), 06Data-Persistence, 10Data-Persistence-Design-Review, 06Growth-Team, and 3 others: Data Persistence Design Review: Improve Tone Suggested Edits newcomer task - https://phabricator.wikimedia.org/T401021#11193561 (10Ottomata) > Only if we also store... [13:33:05] 10Data-Engineering (Q1 FY25/26 July 1st - September 30th), 06Data-Persistence, 10Data-Persistence-Design-Review, 06Growth-Team, and 3 others: Data Persistence Design Review: Improve Tone Suggested Edits newcomer task - https://phabricator.wikimedia.org/T401021#11193631 (10dcausse) >>! In T401021#11193231,... [13:37:53] 10Data-Engineering (Q1 FY25/26 July 1st - September 30th), 10DPE-Mediawiki-Content: MW Content ingest fails with MERGE INTO error - https://phabricator.wikimedia.org/T404975 (10xcollazo) 03NEW [13:44:04] 10Data-Engineering (Q1 FY25/26 July 1st - September 30th), 06Data-Persistence, 10Data-Persistence-Design-Review, 06Growth-Team, and 3 others: Data Persistence Design Review: Improve Tone Suggested Edits newcomer task - https://phabricator.wikimedia.org/T401021#11193689 (10Michael) >>! In T401021#11193561,... [14:02:46] 10Data-Engineering (Q1 FY25/26 July 1st - September 30th), 06Data-Persistence, 10Data-Persistence-Design-Review, 06Growth-Team, and 3 others: Data Persistence Design Review: Improve Tone Suggested Edits newcomer task - https://phabricator.wikimedia.org/T401021#11193749 (10Ottomata) > `hasrecommentation:ton... [14:06:59] 10Data-Engineering (Q1 FY25/26 July 1st - September 30th), 10DPE-Mediawiki-Content: MW Content ingest fails with MERGE INTO error - https://phabricator.wikimedia.org/T404975#11193756 (10xcollazo) 05Open→03In progress p:05Triage→03High [14:08:22] 10Data-Engineering (Q1 FY25/26 July 1st - September 30th), 10DPE-Mediawiki-Content: MW Content ingest fails with MERGE INTO error - https://phabricator.wikimedia.org/T404975#11193763 (10xcollazo) [16:38:08] 10Data-Engineering (Q1 FY25/26 July 1st - September 30th), 10Event-Platform, 10MW-1.45-notes (1.45.0-wmf.19; 2025-09-16), 13Patch-For-Review: EventBus - Add central user id to MediaWiki events - https://phabricator.wikimedia.org/T403664#11194615 (10Ottomata) `mediawiki.ip_reputation.score` stream is now fa... [18:06:01] 10Data-Engineering (Q1 FY25/26 July 1st - September 30th), 06Discovery-Search, 10Wikidata, 10Wikidata-Query-Service, and 2 others: [EPIC] Upgrade flink jobs to java 17 - https://phabricator.wikimedia.org/T404340#11195141 (10dcausse) re eventutilities_python: another point is that booksworm ships with pytho... [18:10:38] 10Data-Engineering (Q1 FY25/26 July 1st - September 30th), 06Growth-Team, 10MediaWiki-Page-derived-data, 06Wikipedia-Android-App-Backlog, 06Wikipedia-iOS-App-Backlog: WE3.3.7 Year in Review and Activity Tab Services - Global Editor Metrics - https://phabricator.wikimedia.org/T403660#11195179 (10Ottomata) [18:16:17] FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-analytics-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=eqiad%2Bprometheus/k8s&var-service=eventgate-analytics-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [18:26:17] RESOLVED: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-analytics-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=eqiad%2Bprometheus/k8s&var-service=eventgate-analytics-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [19:15:12] (03PS3) 10Aleksandar Mastilovic: Address review comments [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1187584 (https://phabricator.wikimedia.org/T401325) [19:15:33] (03CR) 10Aleksandar Mastilovic: "I just corrected both issues." [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1187584 (https://phabricator.wikimedia.org/T401325) (owner: 10Aleksandar Mastilovic) [19:57:15] 10Data-Engineering (Q1 FY25/26 July 1st - September 30th), 10DPE-Mediawiki-Content: MW Content ingest fails with MERGE INTO error - https://phabricator.wikimedia.org/T404975#11195549 (10xcollazo) Upon close inspection, this is indeed a duplicate rows situation similar to T397525. ` spark.sql(""" SELECT count(... [19:57:46] 06Data-Engineering, 06SRE, 06Traffic-Icebox, 10MobileFrontend (Tracking): RFC: Serve mobile and desktop variants through the same URL (unified mobile routing) - https://phabricator.wikimedia.org/T214998#11195548 (10Krinkle) [20:10:57] 10Data-Engineering (Q1 FY25/26 July 1st - September 30th), 06Growth-Team, 10MediaWiki-Page-derived-data, 06Wikipedia-Android-App-Backlog, 06Wikipedia-iOS-App-Backlog: WE3.3.7 Year in Review and Activity Tab Services - Global Editor Metrics - https://phabricator.wikimedia.org/T403660#11195564 (10Ottomata)... [20:17:14] 10Data-Engineering (Q1 FY25/26 July 1st - September 30th), 06Growth-Team, 10MediaWiki-Page-derived-data, 06Wikipedia-Android-App-Backlog, 06Wikipedia-iOS-App-Backlog: WE3.3.7 Year in Review and Activity Tab Services - Global Editor Metrics - https://phabricator.wikimedia.org/T403660#11195601 (10Ottomata)... [20:25:06] 10Data-Engineering (Q1 FY25/26 July 1st - September 30th), 10MediaWiki-Page-derived-data: Global Editor Metrics - Data Pipeline - https://phabricator.wikimedia.org/T405039 (10Ottomata) 03NEW [20:27:02] 10Data-Engineering (Q1 FY25/26 July 1st - September 30th), 06Growth-Team, 10MediaWiki-Page-derived-data, 06Wikipedia-Android-App-Backlog, and 2 others: WE3.3.7 Year in Review and Activity Tab Services - Global Editor Metrics - https://phabricator.wikimedia.org/T403660#11195654 (10Ottomata) [20:27:04] 10Data-Engineering (Q1 FY25/26 July 1st - September 30th), 10MediaWiki-Page-derived-data, 07OKR-Work: Global Editor Metrics - Data Pipeline - https://phabricator.wikimedia.org/T405039#11195655 (10Ottomata) [20:28:52] 10Data-Engineering (Q1 FY25/26 July 1st - September 30th), 10MediaWiki-Page-derived-data, 07OKR-Work: Global Editor Metrics - backfill data - https://phabricator.wikimedia.org/T405040 (10Ottomata) 03NEW [20:34:45] 10Data-Engineering (Q1 FY25/26 July 1st - September 30th), 10MediaWiki-Page-derived-data, 07OKR-Work: Global Editor Metrics - backfill data - https://phabricator.wikimedia.org/T405040#11195687 (10Ottomata) [20:40:02] 10Data-Engineering (Q1 FY25/26 July 1st - September 30th), 10DPE-Mediawiki-Content: MW Content ingest fails with MERGE INTO error - https://phabricator.wikimedia.org/T404975#11195698 (10xcollazo) Pausing all [[ https://airflow.wikimedia.org/home?tags=mediawiki_content | MW Content ]] pipelines. [20:40:34] 10Data-Engineering (Q1 FY25/26 July 1st - September 30th), 10DPE-Mediawiki-Content: MW Content ingest fails with MERGE INTO error - https://phabricator.wikimedia.org/T404975#11195699 (10xcollazo) We will now attempt a fix similar to T397525#10935336. [20:43:22] 10Data-Engineering (Q1 FY25/26 July 1st - September 30th), 06Growth-Team, 10MediaWiki-Page-derived-data, 06Wikipedia-Android-App-Backlog, and 2 others: Global Editor Metrics - HTTP API endpoints - https://phabricator.wikimedia.org/T405041 (10Ottomata) 03NEW [20:43:53] 10Data-Engineering (Q1 FY25/26 July 1st - September 30th), 06Data-Persistence, 10Data-Persistence-Design-Review: Global Editor Metrics - Data Persistence Design Review: - https://phabricator.wikimedia.org/T401260#11195718 (10Ottomata) [20:43:57] 10Data-Engineering (Q1 FY25/26 July 1st - September 30th), 06Data-Persistence, 10Data-Persistence-Design-Review: Global Editor Metrics - Data Persistence Design Review - https://phabricator.wikimedia.org/T401260#11195719 (10Ottomata) [20:56:16] 10Data-Engineering (Q1 FY25/26 July 1st - September 30th), 06Data-Persistence, 10Data-Persistence-Design-Review, 06Growth-Team, and 3 others: Data Persistence Design Review: Improve Tone Suggested Edits newcomer task - https://phabricator.wikimedia.org/T401021#11195747 (10Ottomata) a:05Ottomata→03achou... [22:07:23] (03PS1) 10Aleksandar Mastilovic: Update changelog.md for v0.3.2 [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1189588 (https://phabricator.wikimedia.org/T404735) [22:55:51] 10Data-Engineering (Q1 FY25/26 July 1st - September 30th), 06Data-Persistence, 10Data-Persistence-Design-Review, 06Growth-Team, and 3 others: Data Persistence Design Review: Improve Tone Suggested Edits newcomer task - https://phabricator.wikimedia.org/T401021#11195972 (10KStoller-WMF) >>! In T401021#11193...