[04:28:26] 06Data-Engineering, 06Data-Engineering-Radar, 06Commons, 06Data-Persistence, and 5 others: Migrate file tables to a modern layout (image/oldimage; file/filerevision; add primary keys) - https://phabricator.wikimedia.org/T28741#11621467 (10Nurzebetty3) [06:53:15] 06Data-Engineering, 06Data-Engineering-Radar, 06Commons, 06Data-Persistence, and 5 others: Migrate file tables to a modern layout (image/oldimage; file/filerevision; add primary keys) - https://phabricator.wikimedia.org/T28741#11621548 (10valerio.bozzolan) [09:26:18] 06Data-Engineering, 06Data-Engineering-Radar, 06Commons, 06Data-Persistence, and 5 others: Migrate file tables to a modern layout (image/oldimage; file/filerevision; add primary keys) - https://phabricator.wikimedia.org/T28741#11621880 (10Ciencia_Al_Poder) [09:26:40] 06Data-Engineering, 06Data-Engineering-Radar, 06Commons, 06Data-Persistence, and 5 others: Migrate file tables to a modern layout (image/oldimage; file/filerevision; add primary keys) - https://phabricator.wikimedia.org/T28741#11621881 (10Ciencia_Al_Poder) [09:27:01] 06Data-Engineering, 06Data-Engineering-Radar, 06Commons, 06Data-Persistence, and 5 others: Migrate file tables to a modern layout (image/oldimage; file/filerevision; add primary keys) - https://phabricator.wikimedia.org/T28741#11621882 (10Ciencia_Al_Poder) [09:27:20] 06Data-Engineering, 10MediaWiki-extensions-EventLogging, 07Essential-Work, 13Patch-For-Review, and 2 others: Decouple mw.xLab.getInstrument() and mw.eventLog.newInstrument() - https://phabricator.wikimedia.org/T408094#11621884 (10phuedx) a:03phuedx [09:27:22] 06Data-Engineering, 06Data-Engineering-Radar, 06Commons, 06Data-Persistence, and 5 others: Migrate file tables to a modern layout (image/oldimage; file/filerevision; add primary keys) - https://phabricator.wikimedia.org/T28741#11621883 (10Ciencia_Al_Poder) [09:36:58] 06Data-Engineering, 10MediaWiki-extensions-EventLogging, 07Essential-Work, 13Patch-For-Review, and 2 others: Decouple mw.xLab.getInstrument() and mw.eventLog.newInstrument() - https://phabricator.wikimedia.org/T408094#11621955 (10phuedx) [09:50:51] 06Data-Engineering, 06Discovery-Search, 10Wikidata, 10Wikidata-Query-Service, and 2 others: [EPIC] Upgrade flink jobs to java 17 - https://phabricator.wikimedia.org/T404340#11622018 (10trueg) The rdf-streaming-updater is now running on top of java17 with flink 1.20.2 on both codfw and eqiad. So far it look... [11:00:42] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 13Patch-For-Review: Reduce main Airflow DB size and consider splitting heavy workloads into separate instances - https://phabricator.wikimedia.org/T411992#11622299 (10Antoine_Quhen) Deployment process of db_cleanup dag patch: * Today: * merge the db_c... [12:18:23] 06Data-Engineering, 10Wikidata, 10Wikidata Analytics: WMDE test Airflow instance can't egress out to external URLs - https://phabricator.wikimedia.org/T417630#11622491 (10AndrewTavis_WMDE) [12:19:43] 06Data-Engineering: Airflow devenv (WMDE) cannot see webproxy - https://phabricator.wikimedia.org/T417633 (10awight) 03NEW [12:28:10] 06Data-Engineering, 10Wikidata, 10Wikidata Analytics: WMDE test Airflow instance can't egress out to external URLs - https://phabricator.wikimedia.org/T417630#11622527 (10AndrewTavis_WMDE) [12:57:30] 06Data-Engineering, 10Wikidata, 10Wikidata Analytics: WMDE test Airflow instance can't egress out to external URLs - https://phabricator.wikimedia.org/T417630#11622597 (10AndrewTavis_WMDE) [12:59:41] 06Data-Engineering, 10Wikidata, 10Wikidata Analytics: WMDE test Airflow instance can't egress out to external URLs - https://phabricator.wikimedia.org/T417630#11622601 (10AndrewTavis_WMDE) [14:48:16] (03CR) 10Ejegg: Add first_campaign_status_code for banner activity (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1238821 (https://phabricator.wikimedia.org/T414478) (owner: 10Ejegg) [15:24:41] 06Data-Engineering, 10Wikidata, 10Wikidata Analytics: WMDE test Airflow instance can't egress out to external URLs - https://phabricator.wikimedia.org/T417630#11623378 (10AndrewTavis_WMDE) Note from a [discussion of this on Slack](https://wikimedia.slack.com/archives/C055QGPTC69/p1771331430090369): - Adding... [15:31:19] 06Data-Engineering, 10Wikidata, 10Wikidata Analytics: WMDE test Airflow instance can't egress out to external URLs - https://phabricator.wikimedia.org/T417630#11623406 (10AndrewTavis_WMDE) For now the above method will allow us to test DAGs that include `SparkSqlOperator`. Assuming that this behavior is a bu... [15:52:52] 06Data-Engineering, 06Data-Engineering-Radar, 06Data-Platform-SRE: Requesting Kerberos access for hmonroy - https://phabricator.wikimedia.org/T416729#11623495 (10Ottomata) It looks like kerberos access was granted: https://phabricator.wikimedia.org/T417459#11619232 Will close this as duplicate. Good luck! [15:53:30] 06Data-Engineering, 06Data-Engineering-Radar, 06Data-Platform-SRE: Requesting Kerberos access for hmonroy - https://phabricator.wikimedia.org/T416729#11623497 (10Ottomata) →14Duplicate dup:03T417459 [16:33:05] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 10AQS2.0: Introduce a new AQS endpoint to expose video plays - https://phabricator.wikimedia.org/T415202#11623785 (10Snwachukwu) a:03Snwachukwu [16:42:10] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 06Product-Analytics, 13Patch-For-Review: Propagate field descriptions from event schemas to Hive event tables and into DataHub - https://phabricator.wikimedia.org/T307040#11623842 (10Ottomata) ^ Ah, spark-sql queries fail too. SHOW CREATE TABLE works... [16:43:12] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 06Product-Analytics, 13Patch-For-Review: Propagate field descriptions from event schemas to Hive event tables and into DataHub - https://phabricator.wikimedia.org/T307040#11623869 (10Ottomata) At this point, we could - update and use my old MR but it... [16:51:37] (03CR) 10Ottomata: [C:03+1] "Nice, LGTM." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1239200 (https://phabricator.wikimedia.org/T416481) (owner: 10Snwachukwu) [17:15:18] 06Data-Engineering, 06Data-Engineering-Radar, 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Grant Access to analytics-privatedata-users for AJAVED-WMF - https://phabricator.wikimedia.org/T416922#11624120 (10Dzahn) [17:24:06] 06Data-Engineering, 06Data-Engineering-Radar, 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Grant Access to analytics-privatedata-users for AJAVED-WMF - https://phabricator.wikimedia.org/T416922#11624224 (10Dzahn) [x] L3 has been signed [x] group approver not needed for WMF staff (confirmed staff sta... [17:24:45] 06Data-Engineering, 06Data-Engineering-Radar, 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Grant Access to analytics-privatedata-users for AJAVED-WMF - https://phabricator.wikimedia.org/T416922#11624225 (10Dzahn) @ccasilli Do you approve of this access request for Aiman? thanks! [17:25:42] 06Data-Engineering, 06Data-Engineering-Radar, 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Grant Access to analytics-privatedata-users for AJAVED-WMF - https://phabricator.wikimedia.org/T416922#11624231 (10Dzahn) a:03ccasilli [17:28:12] 06Data-Engineering, 06Movement-Insights: Investigate and repair pageviews and unique devices spike starting in Nov 2025 - https://phabricator.wikimedia.org/T416933#11624241 (10GGoncalves-WMF) [17:31:20] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 10Data Pipelines, 10Wikidata, 10Wikidata Analytics: NEW FEATURE REQUEST: sqoop (all) user properties from mariadb to wmf_raw.mediawiki_user_properties - https://phabricator.wikimedia.org/T323456#11624256 (10Ahoelzl) a:03Snwachukwu [17:36:34] 06Data-Engineering, 06Data-Engineering-Radar, 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Grant Access to analytics-privatedata-users for AJAVED-WMF - https://phabricator.wikimedia.org/T416922#11624292 (10AJaved-WMF) [17:37:39] 06Data-Engineering, 06Data-Engineering-Radar, 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Grant Access to analytics-privatedata-users for AJAVED-WMF - https://phabricator.wikimedia.org/T416922#11624298 (10AJaved-WMF) Hi @Dzahn apologies for not making the edits earlier, I've done so now. I also che... [17:52:30] 06Data-Engineering, 06Data-Engineering-Radar, 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Grant Access to analytics-privatedata-users for AJAVED-WMF - https://phabricator.wikimedia.org/T416922#11624378 (10Dzahn) Hi! Could you make her leave a quick comment here on the ticket? Thanks [18:13:02] 06Data-Engineering, 06Data-Engineering-Radar, 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Grant Access to analytics-privatedata-users for AJAVED-WMF - https://phabricator.wikimedia.org/T416922#11624507 (10ccasilli) Hi there, thanks, all. Aiman you should have access now! [18:20:32] 06Data-Engineering, 06Data-Engineering-Radar, 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Grant Access to analytics-privatedata-users for AJAVED-WMF - https://phabricator.wikimedia.org/T416922#11624570 (10Aklapper) @AJaved-WMF Please also [link your LDAP account to your Phabricator account](https:/... [18:23:14] 06Data-Engineering, 06Data-Engineering-Radar, 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Grant Access to analytics-privatedata-users for AJAVED-WMF - https://phabricator.wikimedia.org/T416922#11624577 (10AJaved-WMF) Done [18:49:56] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 06Product-Analytics, 13Patch-For-Review: Propagate field descriptions from event schemas to Hive event tables and into DataHub - https://phabricator.wikimedia.org/T307040#11624714 (10xcollazo) > Just drop it and push for Refine to Iceberg where this i... [18:56:16] (03CR) 10Joal: Add first_campaign_status_code for banner activity (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1238821 (https://phabricator.wikimedia.org/T414478) (owner: 10Ejegg) [18:59:42] (03CR) 10Xcollazo: Adapt imagelinks pipeline and consumers for imagelink normalization (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1239200 (https://phabricator.wikimedia.org/T416481) (owner: 10Snwachukwu) [19:21:53] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 06Product-Analytics, 13Patch-For-Review: Propagate field descriptions from event schemas to Hive event tables and into DataHub - https://phabricator.wikimedia.org/T307040#11624804 (10Ottomata) For Hive tables, we still use Hive JDBC to evolve the tabl... [19:30:34] (03PS7) 10Ejegg: Add first_campaign_status_code for banner activity [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1238821 (https://phabricator.wikimedia.org/T414478) [19:31:18] (03CR) 10Snwachukwu: Adapt imagelinks pipeline and consumers for imagelink normalization (032 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1239200 (https://phabricator.wikimedia.org/T416481) (owner: 10Snwachukwu) [19:37:32] 06Data-Engineering: Perform a one-time clean up of retained data sets in event_sanitize - https://phabricator.wikimedia.org/T417694 (10xcollazo) 03NEW [19:38:10] 06Data-Engineering: Perform a one-time clean up of retained data sets in event_sanitize - https://phabricator.wikimedia.org/T417694#11624897 (10xcollazo) [19:57:08] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 06Product-Analytics, 13Patch-For-Review: Propagate field descriptions from event schemas to Hive event tables and into DataHub - https://phabricator.wikimedia.org/T307040#11624973 (10xcollazo) Got it. +1 to live with this for Hive, and to invest in m... [20:21:45] (03CR) 10Ejegg: "OK Joal, i've used is_ as that's conventional for booleans, and I added the parens" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1238821 (https://phabricator.wikimedia.org/T414478) (owner: 10Ejegg) [20:54:55] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 13Patch-For-Review: File Export files with revision ranges do not show end revision in filename - https://phabricator.wikimedia.org/T416176#11625207 (10xcollazo) (still need to deploy Airflow patch) [21:13:29] 06Data-Engineering, 06Data-Engineering-Radar, 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Grant Access to analytics-privatedata-users for AJAVED-WMF - https://phabricator.wikimedia.org/T416922#11625255 (10Dzahn) [21:14:10] 06Data-Engineering, 06Data-Engineering-Radar, 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Grant Access to analytics-privatedata-users for AJAVED-WMF - https://phabricator.wikimedia.org/T416922#11625262 (10Dzahn) [21:15:04] 06Data-Engineering, 06Data-Engineering-Radar, 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Grant Access to analytics-privatedata-users for AJAVED-WMF - https://phabricator.wikimedia.org/T416922#11625266 (10Dzahn) @AJaved-WMF You have been added to the requested group. Please give it about 30 minutes... [22:07:15] 06Data-Engineering, 06Data-Engineering-Radar, 06SRE, 10SRE-Access-Requests: Grant Access to analytics-privatedata-users for AJAVED-WMF - https://phabricator.wikimedia.org/T416922#11625413 (10Dzahn) a:05ccasilli→03AJaved-WMF [23:00:52] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th): OpsWeek: wikidatawiki fails to export due to OOM on partitionByXMLFileBoundaries - https://phabricator.wikimedia.org/T416642#11625531 (10Ahoelzl) Great job @xcollazo and @JAllemandou on debugging and fixing this intricate pipeline behavior.